Image recognition via brain-computer collabration with variable image presentation times

ABSTRACT

This application provides an image recognition method and device. The image recognition method includes: setting a presentation time sequence corresponding to an image sequence includes N images, the presentation time sequence includes unequal presentation times, a difference between any two presentation times of the unequal presentation times is k×Δ, k is a positive integer, and Δ is a preset time period value; processing the image sequence by using a computer vision algorithm, to obtain a computer vision signal corresponding to each image in the image sequence; obtaining a feedback signal that is corresponding to each image in the image sequence generated when an observation object watches the image sequence displayed in the presentation time sequence; and fusing, for each image in the image sequence, a corresponding computer vision signal and a corresponding feedback signal to obtain a target recognition signal of each image in the image sequence.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Application No.PCT/CN2019/076657, filed on Mar. 1, 2019, which claims priority toChinese Patent Application No. 201810174915.2, filed on Mar. 2, 2018.The disclosures of the aforementioned applications are hereinincorporated by reference in their entireties.

TECHNICAL FIELD

Embodiments of the present invention relate to the field of informationtechnologies, and in particular, to an image recognition method anddevice, and an image presentation time adjustment method and device.

BACKGROUND

In the current information age, people share abundant informationresources, but also often encounter the problem of “informationoverload” or “information explosion”. How to efficiently select the mostneeded information from the massive information resources is animportant topic in the coming information era. In the image field, imagerecognition is one of the most concerned problems. Image recognition maybe implemented by using a computer vision algorithm. The computer visionalgorithm may be a conventional image detection algorithm, or may be adeep learning algorithm based on an artificial neural network. Theconventional image detection algorithm extracts image features from animage area, and classifies images based on whether an image is a targetimage according to the image classification algorithm. The deep learningalgorithm based on the artificial neural network may train an initialconvolutional neural network by using a training sample, adjust aparameter in the initial convolutional neural network to converge anerror of image recognition, so as to construct a new convolutionalneural network, and predict a probability that an image is a targetimage by using the new convolutional neural network, so as to performimage recognition.

Both the conventional target detection algorithm and the deep learningalgorithm based on the artificial neural network have the followingdisadvantages: First, it may be difficult to obtain training data of aspecific type, which causes unbalanced distribution of training samples.Second, noise of training data is large, which causes a large error ofthe algorithm. In addition, some features of the image, for example, ahigh-order semantic feature, is difficult to extract. Compared with thecomputer vision algorithm, human brain has abundant cognition andapriori knowledge. Extracting a feature by human brain can beindependent of the problems such as an amount of training data and theunbalanced sample distribution. In addition, the human brain oftenexhibits strong stability even under the impact of noise. In addition,the human brain's experience, and high-level semantic understanding andinference ability can also be used to find some obscure high-levelfeatures. However, the human brain has some disadvantages in targetimage recognition, for example, relatively low efficiency. Therefore,persons skilled in the art can think of combining the advantages of thehuman brain and the computer, and performing image recognition throughbrain computer coordination, that is, collaboration between the humanbrain and the computer vision algorithm.

When the brain collaborates with the computer on target imagerecognition, an image sequence based on a rapid serial visualpresentation (RSVP) paradigm may be used as an external stimulus of thehuman brain. When a person observes the image sequence,electroencephalogram (EEG) signals of the human brain that are obtainedwhen the person observes the target image and a common image havedifferent features. Electroencephalogram signals obtained when the humanbrain observes an image sequence can be collected and analyzed, and animage feature of an image in the image sequence can be collected byusing the computer vision algorithm. For each image in the imagesequence, whether the image is a target image may be recognized based onan electroencephalogram signal and an image feature. Currently, a timeinterval between images in an image sequence based on RSVP is determinedaccording to experience or an experiment. However, because a human brainis prone to fatigue and attention resources of the human brain arelimited, a miss detection rate of brain-computer collaboration imagerecognition is still relatively high, resulting in relatively lowefficiency of brain-computer collaboration image recognition.

SUMMARY

Embodiments of this application disclose a brain-computer combinationimage recognition method and device based on image sequencepresentation, and an image presentation time adjustment method anddevice, so as to improve efficiency of brain-computer combination imagerecognition.

According to a first aspect, an embodiment of this application providesa brain-computer combination image recognition method based on imagesequence presentation, including: setting a presentation time sequencecorresponding to an image sequence, where the image sequence includes Nimages, N is a positive integer, the presentation time sequence includesa presentation time of each image in the image sequence, a presentationtime of an image i is used to indicate a time period from a presentationstart moment of the image i to a presentation start moment of a nextadjacent image, the image i is any image in the image sequence, thepresentation time sequence includes at least two unequal presentationtimes, a difference between any two presentation times of the at leasttwo unequal presentation times is k×Δ, k is a positive integer, and Δ isa preset time period value; processing the image sequence by using acomputer vision algorithm, to obtain a computer vision signalcorresponding to each image in the image sequence; obtaining a feedbacksignal that is generated when an observation object watches the imagesequence displayed in the presentation time sequence and thatcorresponds to each image in the image sequence, where the feedbacksignal is used to indicate a reaction of the observation object to thewatched image; and fusing, for each image in the image sequence, acorresponding computer vision signal and a corresponding feedback signalto obtain a target recognition signal of each image in the imagesequence, where the target recognition signal is used for imagerecognition.

The at least two unequal presentation times are used to improve accuracyof recognizing each image in the image sequence by the observationobject. Δ may be a value between 10 ms to 100 ms. In an embodiment, Δmay be a value between 50 ms to 100 ms.

In an embodiment, the image sequence may be from a camera device, andbefore the setting a presentation time sequence corresponding to animage sequence, the method further includes: receiving M images from thecamera device, where M is an integer greater than 1; and selecting Nimages from the M images as the image sequence, where N is less than orequal to M.

In an embodiment, the setting a presentation time sequence correspondingto an image sequence includes: determining a corresponding presentationtime for each image in the image sequence based on a duration impactparameter, to obtain the presentation time sequence corresponding to theimage sequence; where the duration impact parameter includes at leastone of the first recognition probability and a fatigue state parameter,the first recognition probability is used to indicate a probability,obtained by using the computer vision algorithm, that an image includesa preset image feature, the fatigue state parameter is used to indicatea fatigue degree of the observation object when the observation objectobserves an image, the presentation time is inversely correlated withthe first recognition probability, and the presentation time ispositively correlated with the fatigue state parameter. A presentationtime of any image i in the image sequence is set based on at least oneof a first recognition probability and a fatigue state parameter. Ahigher probability that the computer vision algorithm recognizes thatthe image i includes the preset image feature indicates that the imagerecognition device may set a longer brain recognition duration. Moreattention resources in a time dimension of the observation object areallocated to an image with greater uncertainty. This can reduce a missdetection rate of image recognition and improve efficiency ofbrain-computer collaboration image recognition. In addition, a largerfatigue state parameter of the observation object indicates that theobservation object is more fatigued. The observation object needs arelatively long observation time to recognize whether the image iincludes the preset image feature. Therefore, a relatively longpresentation time of the image i is set. Determining the presentationtime of the image i based on a fatigue degree of the observation objectcan reduce miss detection cases caused by brain fatigue of theobservation object, thereby reducing the miss detection rate.

In an embodiment, before the determining a corresponding presentationtime for each image in the image sequence based on a duration impactparameter, the method further includes: predicting, according to afatigue rule, the fatigue state parameter corresponding to each image inthe image sequence, where the fatigue rule is used to indicate a changerule of a fatigue degree of the observation object. A fatigue stateparameter corresponding to any image i in the image sequence when theobservation object observes the image i is predicted by using thefatigue rule, and the presentation time sequence corresponding to theimage sequence may be preset. When the brain and the computer arecombined to perform image recognition on the image, a process ofdetermining a presentation time of the image in the image sequence doesnot need to be executed. This can reduce data load of parallelprocessing performed by the image recognition device during recognitionof the image in the image sequence, thereby reducing a miss detectionrate of brain-computer combination image recognition.

The fatigue state parameter corresponding to the image is a fatiguestate parameter of the observation object when the observation objectwatches the image.

In an embodiment, the duration impact parameter includes the fatiguestate parameter and the first recognition probability. Before thedetermining a corresponding presentation time for each image in theimage sequence based on a duration impact parameter, to obtain thepresentation time sequence corresponding to the image sequence, themethod further includes: processing the image sequence by using thecomputer vision algorithm, to obtain the first recognition probabilityof each image in the image sequence; and predicting, according to afatigue rule, the fatigue state parameter corresponding to each image inthe image sequence, where the fatigue rule is used to indicate a changerule of a fatigue degree of the observation object. The determining acorresponding presentation time for each image in the image sequencebased on a duration impact parameter, to obtain the presentation timesequence corresponding to the image sequence includes: for each image inthe image sequence, determining a corresponding presentation time basedon the first recognition probability and the fatigue state parameter, toobtain the presentation time sequence corresponding to the imagesequence.

In an embodiment, the duration impact parameter includes the fatiguestate parameter. Before the determining a corresponding presentationtime for each image in the image sequence based on a duration impactparameter, to obtain the presentation time sequence corresponding to theimage sequence, the method further includes: predicting, according to afatigue rule, the fatigue state parameter corresponding to each image inthe image sequence, where the fatigue rule is used to indicate a changerule of a fatigue degree of the observation object. The determining acorresponding presentation time for each image in the image sequencebased on a duration impact parameter, to obtain the presentation timesequence corresponding to the image sequence includes: for each image inthe image sequence, determining a corresponding presentation time basedon the fatigue state parameter, to obtain the presentation time sequencecorresponding to the image sequence.

In an embodiment, the duration impact parameter includes the firstrecognition probability. Before the determining a correspondingpresentation time for each image in the image sequence based on aduration impact parameter, to obtain the presentation time sequencecorresponding to the image sequence, the method further includes:processing the image sequence by using the computer vision algorithm, toobtain the first recognition probability of each image in the imagesequence. The determining a corresponding presentation time for eachimage in the image sequence based on a duration impact parameter, toobtain the presentation time sequence corresponding to the imagesequence includes: for each image in the image sequence, determining acorresponding presentation time based on the first recognitionprobability, to obtain the presentation time sequence corresponding tothe image sequence.

In an embodiment, the obtaining a feedback signal that is generated whenan observation object watches the image sequence displayed in thepresentation time sequence and that corresponds to each image in theimage sequence includes: in a process of displaying the image sequencein the presentation time sequence, obtaining the fatigue state parametercorresponding to an image j, and adjusting, based on the fatigue stateparameter corresponding to the image j, a presentation time, in thepresentation time sequence, corresponding to an image to be displayedafter the image j in the image sequence, where the image j is any imagein the image sequence. First, the presentation time sequencecorresponding to the image sequence is determined according to thecomputer vision algorithm. When the image sequence is displayed insequence according to the presentation time sequence, when the brain andthe computer are combined to recognize the image in the image sequence,a process of determining the presentation time sequence of the imagesequence does not need to be executed. This can reduce data load ofparallel processing performed by the image recognition device duringrecognition of the image j, thereby reducing a miss detection rate ofbrain-computer combination image recognition. Second, a fatigue stateparameter corresponding to the observation object when the observationobject observes an image in the image sequence may be detected in realtime by using a fatigue detection apparatus, and a presentation time ofthe image is adjusted based on the real-time fatigue state parameter.The fatigue state parameter can assist in correcting the presentationtime of the image, and can reduce miss detection cases caused by brainfatigue of the observation object, thereby reducing the miss detectionrate.

In an embodiment, the fatigue state parameter corresponding to the imagej may be obtained through prediction according to the fatigue rule.

In an embodiment, the fatigue state parameter corresponding to the imagej may be obtained in real time through prediction by using a sensor. Theobtaining the fatigue state parameter corresponding to the image jincludes: obtaining the fatigue state parameter based on fatigue stateinformation that is sent by a sensor and that is obtained when theobservation object watches the image j.

In an embodiment, the image recognition device may also obtain thepresentation time sequence corresponding to the image sequence based onthe fatigue state parameter, and then adjust the presentation time ofthe image in the image sequence in real time based on the firstrecognition probability. Specifically, the duration impact parameterincludes the fatigue state parameter. The obtaining a feedback signalthat is generated when an observation object watches the image sequencedisplayed in the presentation time sequence and that corresponds to eachimage in the image sequence includes: in a process of displaying theimage sequence in the presentation time sequence, obtaining the firstrecognition probability corresponding to an image j, and adjusting,based on the first recognition probability corresponding to the image j,a presentation time, in the presentation time sequence, corresponding toan image to be displayed after the image j in the image sequence, wherethe image j is any image in the image sequence.

In an embodiment, the fatigue rule is used to indicate a change rule ofa fatigue degree of the observation object based on a quantity of imagesobserved by the observation object.

In an embodiment, the fatigue rule is used to indicate a change rule ofthe fatigue degree of the observation object based on duration spent bythe observation object for image observation.

In an embodiment, the fatigue rule includes a second mapping table, andthe predicting, according to a fatigue rule, the fatigue state parametercorresponding to each image in the image sequence includes: finding,from the second mapping table according to a quantity of imagesdisplayed before each image in the image sequence, the fatigue stateparameter corresponding to each image in the image sequence. In theimage sequence, a quantity of images displayed before a specific imageis a quantity of observed images corresponding to the specific image,and the second mapping table includes a plurality of quantities ofobserved images and fatigue state parameters corresponding to theplurality of quantities of observed images.

In an embodiment, the fatigue rule includes a second mapping table, andthe predicting, according to the fatigue rule, a fatigue state parametercorresponding to each image in the image sequence includes: predicting,according to a quantity S of images displayed before each image in theimage sequence, a duration spent by the observation object for imageobservation when each image in the image sequence is being observed,where a duration spent by the observation object for image observationwhen an image is being observed is t=S×ts, ts is a predicted averagepresentation time of each image in the image sequence; and finding, fromthe second mapping table, the fatigue state parameter corresponding toeach image in the image sequence according to the duration spent by theobservation object for image observation when each image in the imagesequence is being observed. The second mapping table includes aplurality of durations spent by the observation object for imageobservation and fatigue state parameters corresponding to the pluralityof durations spent by the observation object for image observation.

In an embodiment, the fatigue rule is a fitting formula for the fatiguestate parameter and the quantity of images observed by the observationobject.

In an embodiment, the fatigue rule is a fitting formula for the fatiguestate parameter and the duration spent by the observation object forimage observation.

In an embodiment, the fatigue rule is an objective law related to theobservation object.

In an embodiment, the fatigue rule is obtained by training one or moreobservation objects by using a plurality of samples, and each sample inthe plurality of samples is a combination of a quantity of observedimages and a fatigue state parameter; or each sample in the plurality ofsamples is a combination of a duration spent for image observation and afatigue state parameter.

In an embodiment, the determining a corresponding presentation time foreach image in the image sequence based on a duration impact parameterincludes: for each image in the image sequence, finding a presentationtime corresponding to the duration impact parameter from a first mappingtable, where the first mapping table includes a plurality of durationimpact parameters and presentation times respectively corresponding tothe plurality of duration impact parameters.

In an embodiment, the duration impact parameter includes the firstrecognition probability. The determining a corresponding presentationtime for each image in the image sequence based on a duration impactparameter includes: obtaining the presentation time of each image in theimage sequence by using the following fitting formula:

${{T(c)} = {\sum\limits_{t = 0}^{n}{a_{j}c^{t}}}};$where

T(c) is the presentation time, c is the first recognition probability, cis a real number satisfying 0≤c≤1, n is an order at which T(c) fits c, nis an integer greater than 0, t is an integer satisfying −n≤t≤n, anda_(t) is a coefficient of c^(t).

In an embodiment, T(c) is obtained after n-order linear fitting isperformed on c by using (c1, T2) and (c2, T1). T1 is a minimumpresentation time threshold, T2 is a maximum presentation timethreshold, c1 is a minimum probability threshold of a recognitionprobability determined by using the computer vision algorithm, and c2 isa maximum probability threshold of a recognition probability determinedby using the computer vision algorithm.

In an embodiment, when the first recognition probability of any image iin the image sequence is greater than or equal to c2, the firstrecognition probability is used to determine that the image i includesthe preset image feature. When the first recognition probability of theimage i is less than or equal to c1, the first recognition probabilityis used to determine that the image i does not include the preset imagefeature.

In an embodiment, the duration impact parameter includes the fatiguestate parameter. The determining a corresponding presentation time foreach image in the image sequence based on a duration impact parameterincludes: obtaining the presentation time of each image in the imagesequence by using the following fitting formula:

${{T(f)} = {\sum\limits_{k = 0}^{m}{a_{k}f^{k}}}};$where

T(f) is the presentation time, f is the fatigue state parameter, m is anorder at which T(f) fits f, m is a positive integer greater than 0, k isan integer satisfying −m≤k≤m, and a_(k) is a coefficient of f^(k).

In an embodiment, T(f) is obtained after n-order linear fitting isperformed on f by using (f1, T1) and (f2, T2). T1 is a minimumpresentation time threshold, T2 is a maximum presentation timethreshold, f1 is a minimum fatigue threshold, and f2 is a maximumfatigue threshold.

In an embodiment, when a fatigue state parameter corresponding to anyimage i in the image sequence is greater than or equal to f2, display ofthe image sequence is controlled to be stopped. After the observationobject has a rest for a period of time, in when the fatigue stateparameter corresponding to the image i is less than or equal to f1,display the image sequence starts again.

In an embodiment, the duration impact parameter includes the firstrecognition probability. The determining a corresponding presentationtime for each image in the image sequence based on a duration impactparameter includes: obtaining the presentation time of each image in theimage sequence by using the following fitting formula:

${{T\left( {c,f} \right)} = {\sum\limits_{t = 0}^{n}{\sum\limits_{k = 0}^{m}{a_{t,k}c^{t}f^{k}}}}};$where

T(c, f) is the presentation time, c is the first recognitionprobability, f is the fatigue state parameter, m is an order at whichT(c, f) fits f, n is an order at which T(c, f) fits c, both n and m arepositive integers greater than 0, t is an integer satisfying −n≤t≤n, kis an integer satisfying −m≤k≤m, c is a real number satisfying 0≤c≤1,and a_(t,k) is a coefficient of c^(t)f^(k).

In an embodiment, T(c, f) is obtained after linear fitting is performedon c and f by using (c1, T2), (c2, T1), (f1, T1), and (c2, T2). T1 is aminimum presentation time threshold, T2 is a maximum presentation timethreshold, c1 is a minimum probability threshold of a recognitionprobability determined by using the computer vision algorithm, c2 is amaximum probability threshold of a recognition probability determined byusing the computer vision algorithm, f1 is a minimum fatigue threshold,and f2 is a maximum fatigue threshold.

In an embodiment, when the first recognition probability of any image iin the image sequence is greater than or equal to c2, the firstrecognition probability is used to determine that the image i includesthe preset image feature. When the first recognition probability of theimage i is less than or equal to c1, the first recognition probabilityis used to determine that the image i does not include the preset imagefeature. When a fatigue state parameter corresponding to the image i isgreater than or equal to f2, display of the image sequence is stopped.When the fatigue state parameter corresponding to the image i is lessthan or equal to f1, display the image sequence starts again.

In an embodiment, when it is detected that a corresponding fatigue stateparameter obtained when the observation object observes an image q isgreater than or equal to a first fatigue threshold, images to bedisplayed after the image q in the image sequence are controlled not tobe displayed, and an image whose corresponding first recognitionprobability is greater than or equal to a first probability threshold inthe images to be displayed after the image q is obtained, where theimage q is any image in the image sequence. When it is detected that thefatigue state parameter of the observation object is less than or equalto a second fatigue threshold, the image whose first recognitionprobability is greater than or equal to the first probability thresholdin the images to be displayed after the image q is controlled to besequentially displayed. When the fatigue state parameter of theobservation object is greater than or equal to the first fatiguethreshold, that is, when it is detected that the observation object isalready fatigued, image display is suspended, so that the observationobject rests, and images with a relatively high first recognitionprobability are selected in this period. When the observation object hasfinished rest, brain-computer combination image recognition is performedon these images again. After the foregoing process, image recognitionefficiency can be improved.

In an embodiment, there are at least two observation objects. Thefusing, for each image in the image sequence, a corresponding computervision signal and a corresponding feedback signal to obtain a targetrecognition signal of each image in the image sequence includes: fusing,for each image in the image sequence, a corresponding computer visionsignal and at least two corresponding feedback signals to obtain atarget recognition signal of each image in the image sequence. Aplurality of observation objects simultaneously perform brain-computercombination image recognition on an image in the image sequence. Thiscan reduce a random error caused by a subjective reason of anobservation object in a case of one observation object, therebyimproving accuracy of brain-computer combination image recognition.

In an embodiment, the fatigue state parameter includes at least twofatigue state parameters respectively generated when the at least twoobservation objects observe a same image. Fatigue state parameters of aplurality of observation objects are measured and an image presentationtime is determined by using the plurality of fatigue state parameters.In this way, a random error caused by a subjective reason of anobservation object in a case of one observation object can be reduced,and determining of the image presentation time is more accurate.Therefore, the accuracy of brain-computer combination image recognitioncan be improved.

In an embodiment, the fusing, for each image in the image sequence, acorresponding computer vision signal and a corresponding feedback signalto obtain a target recognition signal of each image in the imagesequence includes: determining, for each image in the image sequencebased on at least one of the first recognition probability, the fatiguestate parameter, and the presentation time, a first weight correspondingto each image in the image sequence, where the first weight is a weightused when the corresponding feedback signal is used to determine thetarget recognition signal, the first weight is inversely correlated withthe first recognition probability, the first weight is inverselycorrelated with the fatigue state parameter, and the first weight ispositively correlated with the presentation time; and fusing, for eachimage in the image sequence based on a corresponding first weight, acorresponding computer vision signal and a corresponding feedback signalto obtain the target recognition signal of each image in the imagesequence. When recognition accuracy of the computer vision algorithm ishigher, a fusion weight used by recognition by using the computer visionalgorithm may be increased, and a fusion weight used by brainrecognition of the observation object may be reduced, thereby reducing amiss detection rate. When the fatigue state parameter of the observationobject is larger, the weight of brain recognition of the observationobject may be reduced, and therefore the miss detection rate can bereduced. When an image presentation time is longer, a time forobservation by the observation object is longer, and therefore accuracyof brain recognition of the observation object is higher. In this case,the weight of brain recognition of the observation object may beincreased, so as to reduce a miss detection rate.

In an embodiment, the computer vision signal is a first recognitionprobability determined by using the computer vision algorithm. Beforethe fusing, for each image in the image sequence, a correspondingcomputer vision signal and a corresponding feedback signal to obtain atarget recognition signal of each image in the image sequence, themethod further includes: calculating, for each image in the imagesequence, a second recognition probability of each image in the imagesequence based on a corresponding feedback signal, where the secondrecognition probability is used to indicate a probability that theobservation object determines that the image includes the preset imagefeature; and the fusing, for each image in the image sequence, acorresponding computer vision signal and a corresponding feedback signalto obtain a target recognition signal of each image in the imagesequence includes: calculating, for each image in the image sequence, atarget recognition probability of each image in the image sequence basedon the corresponding first recognition probability and the correspondingsecond recognition probability.

In an embodiment, the computer vision signal is an image featuredetermined by using the computer vision algorithm. Before the fusing,for each image in the image sequence, a corresponding computer visionsignal and a corresponding feedback signal to obtain a targetrecognition signal of each image in the image sequence, the methodfurther includes: determining, for each image in the image sequencebased on a corresponding feedback signal, a feedback signal featurecorresponding to each image in the image sequence; and the fusing, foreach image in the image sequence, a corresponding computer vision signaland a corresponding feedback signal to obtain a target recognitionsignal of each image in the image sequence includes: performing, foreach image in the image sequence, feature fusion on the correspondingimage feature and the corresponding feedback signal feature, to obtain afused feature corresponding to each image in the image sequence; anddetermining, for each image in the image sequence, a target recognitionprobability of each image in the image sequence based on thecorresponding fused feature.

In an embodiment, S images are determined, from the image sequence basedon the target recognition probability of each image in the imagesequence, as images including the preset image feature, where the targetrecognition probabilities of the S images meet a preset condition, and Sis an integer less than or equal to N. The preset condition may be thatthe target recognition probability is greater than or equal to athreshold, or the preset condition may be that the S images are thefirst S images sorted in descending order according to the targetrecognition probabilities when target recognition probabilities ofimages in the image sequence are sorted in descending order.

In an embodiment, images whose corresponding target recognitionprobabilities are between a second probability threshold and a thirdprobability threshold in the image sequence are used as a new imagesequence. The new image sequence may be used to re-execute the methoddescribed in any one of the first aspect and the embodiments of thefirst aspect. Images, in the image sequence, with relatively greatuncertainty of whether the preset image feature is included are re-used,for a plurality of times, as a new image sequence for brain computercombination image recognition. In this way, a suspicious object in theimage sequence can be filtered out, a probability of misjudgment of theimage recognition device is reduced, and accuracy of brain computercombination image recognition can be improved.

In an embodiment, when a target recognition probability of any image inthe image sequence is less than or equal to the second probabilitythreshold, the image recognition device predicts that the image is notan image that includes the preset image feature. When a targetrecognition probability of any image in the image sequence is greaterthan or equal to the third probability threshold, the image recognitiondevice predicts that the image is an image that includes the presetimage feature. The second probability threshold is less than or equal tothe third probability threshold.

In an embodiment, the feedback signal is an electroencephalogram signal.

According to a second aspect, an embodiment of this application providesan image presentation time determining method, including: obtaining aplurality of images; setting a corresponding presentation time for eachimage in the plurality of images based on a duration impact parameter,to obtain a presentation time sequence corresponding to the plurality ofimages, where the duration impact parameter includes at least one of afirst recognition probability and a fatigue state parameter, thepresentation time is inversely correlated with the first recognitionprobability, the presentation time is positively correlated with thefatigue state parameter, the first recognition probability is used toindicate a probability, obtained by using a computer vision algorithm,that an image includes a preset image feature, the fatigue stateparameter is used to indicate a fatigue degree of the observation objectwhen the observation object observes an image, a presentation time of animage i is used to indicate a time period from a presentation startmoment of the image i to a presentation start moment of a next adjacentimage, and the image i is any image in the plurality of images; andoutputting or storing the plurality of images and the presentation timesequence corresponding to the plurality of images.

The plurality of images may form an image sequence.

In an embodiment, the obtaining a plurality of images includes:receiving M images from a camera device, where M is an integer greaterthan 1; and selecting N images from the M images as the image sequence,where N is less than or equal to M.

In an embodiment, the fatigue state parameter corresponding to eachimage in the image sequence is obtained through prediction according toa fatigue rule, and the fatigue rule is used to indicate a change ruleof a fatigue degree of the observation object.

The fatigue state parameter corresponding to the image is a fatiguestate parameter of the observation object when the observation objectwatches the image.

In an embodiment, after the outputting or storing the plurality ofimages and the presentation time sequence corresponding to the pluralityof images, the method further includes: obtaining a feedback signal thatis generated when the observation object watches the plurality of imagesdisplayed in the presentation time sequence and that corresponds to eachimage in the plurality of images, where the feedback signal is used toindicate a reaction of the observation object to the watched image; andfusing, for each image in the image sequence, a corresponding computervision signal and a corresponding feedback signal to obtain a targetrecognition signal of each image in the plurality of images, where thetarget recognition signal is used for image recognition. Thepresentation time of each image in the image sequence is set based on atleast one of a first recognition probability and a fatigue stateparameter. A computer vision signal corresponding to an image is a firstrecognition probability of the image or an image feature of the imagethat is obtained by processing the image by using a computer visionalgorithm. A higher probability that the computer vision algorithmrecognizes that an image includes the preset image feature indicatesthat an image recognition device may set a longer brain recognitionduration. More attention resources in a time dimension of theobservation object are allocated to an image with greater uncertainty.This can reduce a miss detection rate of image recognition and improveefficiency of brain-computer collaboration image recognition. Inaddition, a larger fatigue state parameter of the observation objectindicates that the observation object is more fatigued, and theobservation object needs a relatively long observation time to recognizewhether an image includes the preset image feature. Therefore, arelatively long presentation time of the image is set. Determining thepresentation time of the image based on a fatigue degree of theobservation object can reduce miss detection cases caused by brainfatigue of the observation object, thereby reducing the miss detectionrate.

In an embodiment, the duration impact parameter includes the firstrecognition probability. The obtaining a feedback signal that isgenerated when the observation object watches the plurality of imagesdisplayed in the presentation time sequence and that corresponds to eachimage in the plurality of images includes: in a process of displayingthe plurality of images in the presentation time sequence, obtaining thefatigue state parameter corresponding to an image j, and adjusting,based on the fatigue state parameter corresponding to the image j, apresentation time, in the presentation time sequence, corresponding toan image to be displayed after the image j in the plurality of images,where the image j is any image in the image sequence.

In an embodiment, the fatigue state parameter corresponding to the imagej may be obtained through prediction according to the fatigue rule.

In an embodiment, the fatigue state parameter corresponding to the imagej may be obtained in real time through prediction by using a sensor.

In an embodiment, the fatigue rule is used to indicate a change rule ofa fatigue degree of the observation object based on a quantity of imagesobserved by the observation object.

In an embodiment, the fatigue rule is used to indicate a change rule ofthe fatigue degree of the observation object based on a duration spentby the observation object for image observation.

In an embodiment, the fatigue rule includes a second mapping table.

In an embodiment, the fatigue rule includes a fitting formula for thefatigue state parameter and the quantity of images observed by theobservation object.

In an embodiment, the fatigue rule is a fitting formula for the fatiguestate parameter and the duration spent by the observation object forimage observation.

In an embodiment, the fatigue rule is an objective law related to theobservation object.

A fatigue state parameter corresponding to any image i in the pluralityof images when the observation object observes the image i is predictedby using the fatigue rule, and the presentation time sequencecorresponding to the image sequence may be preset. When the brain andthe computer are combined to perform image recognition on the image, aprocess of determining a presentation time of the image in the imagesequence does not need to be executed. This can reduce data load ofparallel processing performed by the image recognition device duringrecognition of the image in the image sequence, thereby reducing a missdetection rate of brain-computer combination image recognition.

In an embodiment, the fatigue rule is obtained by training one or moreobservation objects by using a plurality of samples, and each sample inthe plurality of samples is a combination of a quantity of observedimages and a fatigue state parameter; or each sample in the plurality ofsamples is a combination of a duration spent for image observation and afatigue state parameter.

In an embodiment, the determining a corresponding presentation time foreach image in the image sequence based on a duration impact parameterincludes: for each image in the image sequence, finding a presentationtime corresponding to the duration impact parameter from a first mappingtable, where the first mapping table includes a plurality of durationimpact parameters and presentation times respectively corresponding tothe plurality of duration impact parameters.

In an embodiment, the duration impact parameter includes the firstrecognition probability. The setting a corresponding presentation timefor each image in the plurality of images based on a duration impactparameter includes: obtaining the presentation time of each image in theplurality of images by using the following fitting formula:

${{T(c)} = {\sum\limits_{t = 0}^{n}{a_{j}c^{t}}}};$where

T(c) is the presentation time, c is the first recognition probability, cis a real number satisfying 0≤c≤1, n is an order at which T(c) fits c, nis an integer greater than 0, t is an integer satisfying −n≤t≤n, anda_(t) is a coefficient of c^(t).

In an embodiment, T(c) is obtained after n-order linear fitting isperformed on c by using (c1, T2) and (c2, T1). T1 is a minimumpresentation time threshold, T2 is a maximum presentation timethreshold, c1 is a minimum probability threshold of a recognitionprobability determined by using the computer vision algorithm, and c2 isa maximum probability threshold of a recognition probability determinedby using the computer vision algorithm.

In an embodiment, the duration impact parameter includes the fatiguestate parameter. The determining a corresponding presentation time foreach image in the plurality of images based on a duration impactparameter includes: obtaining the presentation time of each image in theplurality of images by using the following fitting formula:

${{T(f)} = {\sum\limits_{k = 0}^{m}{a_{k}f^{k}}}};$where

T(f) is the presentation time, f is the fatigue state parameter, m is anorder at which T(f) fits f, m is a positive integer greater than 0, k isan integer satisfying −m≤k≤m, and a_(k) is a coefficient of f^(k).

In an embodiment, T(f) is obtained after n-order linear fitting isperformed on f by using (f1, T1) and (f2, T2). T1 is a minimumpresentation time threshold, T2 is a maximum presentation timethreshold, f1 is a minimum fatigue threshold, and f2 is a maximumfatigue threshold.

In an embodiment, the duration impact parameter includes the firstrecognition probability and the fatigue state parameter. The determininga corresponding presentation time for each image in the plurality ofimages based on a duration impact parameter includes: obtaining thepresentation time of each image in the plurality of images by using thefollowing fitting formula:

${{T\left( {c,f} \right)} = {\sum\limits_{t = 0}^{n}{\sum\limits_{k = 0}^{m}{a_{t,k}c^{t}f^{k}}}}};$where

T(c, f) is the presentation time, c is the first recognitionprobability, f is the fatigue state parameter, m is an order at whichT(c, f) fits f, n is an order at which T(c, f) fits c, both n and m arepositive integers greater than 0, t is an integer satisfying −n≤t≤n, kis an integer satisfying −m≤k≤m, c is a real number satisfying 0≤c≤1,and a_(t,k) is a coefficient of c^(t)f^(k).

In an embodiment, T(c, f) is obtained after linear fitting is performedon c and f by using (c1, T2), (c2, T1), (f1, T1), and (c2, T2). T1 is aminimum presentation time threshold, T2 is a maximum presentation timethreshold, c1 is a minimum probability threshold of a recognitionprobability determined by using the computer vision algorithm, c2 is amaximum probability threshold of a recognition probability determined byusing the computer vision algorithm, f1 is a minimum fatigue threshold,and f2 is a maximum fatigue threshold.

In an embodiment, when it is detected that a corresponding fatigue stateparameter obtained when the observation object observes an image q isgreater than or equal to a first fatigue threshold, images to bedisplayed after the image q in the plurality of images are controllednot to be displayed, and an image whose corresponding first recognitionprobability is greater than or equal to a first probability threshold inthe images to be displayed after the image q is obtained, where theimage q is any image in the plurality of images. When it is detectedthat the fatigue state parameter of the observation object is less thanor equal to a second fatigue threshold, the image whose firstrecognition probability is greater than or equal to the firstprobability threshold in the images to be displayed after the image q iscontrolled to be sequentially displayed. When the fatigue stateparameter of the observation object is greater than or equal to thefirst fatigue threshold, that is, when it is detected that theobservation object is already fatigued, image display is suspended, sothat the observation object rests, and images with a relatively highfirst recognition probability are selected in this period. When theobservation object has finished rest, brain-computer combination imagerecognition is performed on these images again. After the foregoingprocess, image recognition efficiency can be improved.

In an embodiment, there are at least two observation objects. Thefusing, for each image in the plurality of images, a correspondingcomputer vision signal and a corresponding feedback signal to obtain atarget recognition signal of each image in the plurality of imagesincludes: fusing, for each image in the image sequence, a correspondingcomputer vision signal and at least two corresponding feedback signalsto obtain a target recognition signal of each image in the plurality ofimages. A plurality of observation objects simultaneously performbrain-computer combination image recognition on an image in the imagesequence. This can reduce a random error caused by a subjective reasonof an observation object in a case of one observation object, therebyimproving accuracy of brain-computer combination image recognition.

In an embodiment, the fatigue state parameter includes at least twofatigue state parameters respectively generated when the at least twoobservation objects observe a same image. Fatigue state parameters of aplurality of observation objects are measured and an image presentationtime is determined by using the plurality of fatigue state parameters.In this way, a random error caused by a subjective reason of anobservation object in a case of one observation object can be reduced,and determining of the image presentation time is more accurate.Therefore, the accuracy of brain-computer combination image recognitioncan be improved.

In an embodiment, the fusing, for each image in the plurality of images,a corresponding computer vision signal and a corresponding feedbacksignal to obtain a target recognition signal of each image in theplurality of images includes: determining, for each image in theplurality of images based on at least one of the first recognitionprobability, the fatigue state parameter, and the presentation time, afirst weight corresponding to each image in the plurality of images,where the first weight is a weight used when the corresponding feedbacksignal is used to determine the target recognition signal, the firstweight is inversely correlated with the first recognition probability,the first weight is inversely correlated with the fatigue stateparameter, and the first weight is positively correlated with thepresentation time; and fusing, for each image in the plurality of imagesbased on a corresponding first weight, a corresponding computer visionsignal and a corresponding feedback signal to obtain the targetrecognition signal of each image in the image sequence. When recognitionaccuracy of the computer vision algorithm is higher, a fusion weightused by recognition by using the computer vision algorithm may beincreased, and a fusion weight used by brain recognition of theobservation object may be reduced, thereby reducing a miss detectionrate. When the fatigue state parameter of the observation object islarger, the weight of brain recognition of the observation object may bereduced, and therefore the miss detection rate can be reduced. When animage presentation time is longer, a time for observation by theobservation object is longer, and therefore accuracy of brainrecognition of the observation object is higher. In this case, theweight of brain recognition of the observation object may be increased,so as to reduce a miss detection rate.

In an embodiment, the computer vision signal is a first recognitionprobability determined by using the computer vision algorithm. Beforethe fusing, for each image in the plurality of images, a correspondingcomputer vision signal and a corresponding feedback signal to obtain atarget recognition signal of each image in the plurality of images, themethod further includes: calculating, for each image in the plurality ofimages, a second recognition probability of each image in the pluralityof images based on a corresponding feedback signal, where the secondrecognition probability is used to indicate a probability that theobservation object determines that the image includes the preset imagefeature; and the fusing, for each image in the plurality of images, acorresponding computer vision signal and a corresponding feedback signalto obtain a target recognition signal of each image in the plurality ofimages includes: calculating, for each image in the plurality of images,a target recognition probability of each image in the plurality ofimages based on the corresponding first recognition probability and thecorresponding second recognition probability.

In an embodiment, the computer vision signal is an image featuredetermined by using the computer vision algorithm. Before the fusing,for each image in the plurality of images, a corresponding computervision signal and a corresponding feedback signal to obtain a targetrecognition signal of each image in the plurality of images, the methodfurther includes: determining, for each image in the plurality of imagesbased on a corresponding feedback signal, a feedback signal featurecorresponding to each image in the plurality of images; and the fusing,for each image in the plurality of images, a corresponding computervision signal and a corresponding feedback signal to obtain a targetrecognition signal of each image in the plurality of images includes:performing, for each image in the image sequence, feature fusion on thecorresponding image feature and the corresponding feedback signalfeature, to obtain a fused feature corresponding to each image in theplurality of images; and determining, for each image in the plurality ofimages, a target recognition probability of each image in the pluralityof images based on the corresponding fused feature.

In an embodiment, S images are determined, from the plurality of imagesbased on the target recognition probability of each image in theplurality of images, as images including the preset image feature, wherethe target recognition probabilities of the S images meet a presetcondition, and S is an integer less than or equal to N. The presetcondition may be that the target recognition probability is greater thanor equal to a threshold, or the preset condition may be that the Simages are the first S images sorted in descending order according tothe target recognition probabilities when target recognitionprobabilities of images in the image sequence are sorted in descendingorder.

In an embodiment, images whose corresponding target recognitionprobabilities are between a second probability threshold and a thirdprobability threshold in the plurality of images are used as a new groupof a plurality of images, and the new group of a plurality of images maybe used to re-execute the method described in any one of the secondaspect and the embodiments of the second aspect. Images, in the imagesequence, with relatively great uncertainty of whether the preset imagefeature is included are re-used, for a plurality of times, as a newimage sequence for brain computer combination image recognition. In thisway, a suspicious object in the image sequence can be filtered out, aprobability of misjudgment of the image recognition device is reduced,and accuracy of brain computer combination image recognition can beimproved.

In an embodiment, when a target recognition probability of any image inthe plurality of images is less than or equal to the second probabilitythreshold, the image recognition device predicts that the image is notan image that includes the preset image feature. When a targetrecognition probability of any image in the plurality of images isgreater than or equal to the third probability threshold, the imagerecognition device predicts that the image is an image that includes thepreset image feature. The second probability threshold is less than orequal to the third probability threshold.

In an embodiment, the feedback signal is an electroencephalogram signal.

According to a third aspect, an embodiment of this application providesan image presentation time adjustment method, including: obtaining animage sequence based on a rapid serial visual presentation RSVPparadigm, where the image sequence includes a plurality of images, apresentation time is configured for each image in the plurality ofimages, a presentation time of an image i is used to indicate a timeperiod from a presentation start moment of the image i to a presentationstart moment of a next adjacent image, and the image i is any image inthe plurality of images; adjusting the presentation time correspondingto each image in the image sequence based on a corresponding durationimpact parameter for each image in the image sequence, where theduration impact parameter includes at least one of a first recognitionprobability and a fatigue state parameter, the first recognitionprobability is used to indicate a probability, obtained by using acomputer vision algorithm, that an image includes a preset imagefeature, the fatigue state parameter is used to indicate a fatiguedegree of the observation object when the observation object observes animage, the first recognition probability is inversely correlated withthe presentation time, and the fatigue state parameter is positivelycorrelated with the presentation time; and controlling display of theimage sequence based on an adjusted presentation time corresponding toeach image in the image sequence.

In an embodiment, presentation times of the plurality of images beforethe presentation time adjustment starts are equal.

In an embodiment, the obtaining an image sequence based on a rapidserial visual presentation RSVP paradigm includes: receiving M imagesfrom a camera device, where M is an integer greater than 1; andselecting N images from the M images as the image sequence, where N isless than or equal to M.

In an embodiment, the fatigue state parameter corresponding to eachimage in the image sequence is obtained through prediction according toa fatigue rule, and the fatigue rule is used to indicate a change ruleof a fatigue degree of the observation object.

In an embodiment, the fatigue state parameter corresponding to the imageis a fatigue state parameter of the observation object when theobservation object watches the image.

In an embodiment, after the controlling display of the image sequencebased on an adjusted presentation time corresponding to each image inthe image sequence, the method further includes: obtaining a feedbacksignal that is generated when the observation object watches the imagesequence displayed in the presentation time sequence and thatcorresponds to each image in the image sequence, where the feedbacksignal is used to indicate a reaction of the observation object to thewatched image; and fusing, for each image in the image sequence, acorresponding computer vision signal and a corresponding feedback signalto obtain a target recognition signal of each image in the imagesequence, where the target recognition signal is used for imagerecognition. A computer vision signal corresponding to an image is afirst recognition probability of the image or an image feature of theimage that is obtained by processing the image by using a computervision algorithm. A higher probability that the computer visionalgorithm recognizes that an image includes the preset image featureindicates that an image recognition device may set a longer brainrecognition duration. More attention resources in a time dimension ofthe observation object are allocated to an image with greateruncertainty. This can reduce a miss detection rate of image recognitionand improve efficiency of brain-computer collaboration imagerecognition. In addition, a larger fatigue state parameter of theobservation object indicates that the observation object is morefatigued. The observation object needs a relatively long observationtime to recognize whether an image includes the preset image feature.Therefore, a relatively long presentation time of the image is set.Determining the presentation time of the image based on a fatigue degreeof the observation object can reduce miss detection cases caused bybrain fatigue of the observation object, thereby reducing the missdetection rate.

In an embodiment, the duration impact parameter includes the fatiguestate parameter and the first recognition probability. Before theadjusting the presentation time corresponding to each image in the imagesequence based on a corresponding duration impact parameter for eachimage in the image sequence, the method further includes: processing theimage sequence by using the computer vision algorithm, to obtain thefirst recognition probability of each image in the image sequence; andpredicting, according to a fatigue rule, the fatigue state parametercorresponding to each image in the image sequence, where the fatiguerule is used to indicate a change rule of a fatigue degree of theobservation object. The adjusting the presentation time corresponding toeach image in the image sequence based on a corresponding durationimpact parameter for each image in the image sequence includes: for eachimage in the image sequence, adjusting a presentation time of acorresponding image based on the corresponding first recognitionprobability and the corresponding fatigue state parameter, to obtain anadjusted presentation time sequence corresponding to the image sequence.The presentation time sequence corresponding to the image sequence maybe adjusted in advance. When the brain and the computer are combined toperform image recognition on the image, a process of adjusting apresentation time of the image in the image sequence does not need to beexecuted. This can reduce data load of parallel processing performed bythe image recognition device during recognition of the image in theimage sequence, thereby reducing a miss detection rate of brain-computercombination image recognition.

In an embodiment, the duration impact parameter includes the firstrecognition probability. Before the adjusting the presentation timecorresponding to each image in the image sequence based on acorresponding duration impact parameter for each image in the imagesequence, the method further includes: processing the image sequence byusing the computer vision algorithm, to obtain the first recognitionprobability of each image in the image sequence. The adjusting thepresentation time corresponding to each image in the image sequencebased on a corresponding duration impact parameter for each image in theimage sequence includes: for each image in the image sequence, adjustingthe presentation time corresponding to each image in the image sequencebased on the corresponding first recognition probability.

In an embodiment, the duration impact parameter includes the fatiguestate parameter. Before the adjusting the presentation timecorresponding to each image in the image sequence based on acorresponding duration impact parameter for each image in the imagesequence, the method further includes: predicting, according to afatigue rule, the fatigue state parameter corresponding to each image inthe image sequence, where the fatigue rule is used to indicate a changerule of a fatigue degree of the observation object. The adjusting thepresentation time corresponding to each image in the image sequencebased on a corresponding duration impact parameter for each image in theimage sequence includes: for each image in the image sequence, adjustingthe presentation time corresponding to each image in the image sequencebased on the corresponding fatigue state parameter.

In an embodiment, the adjusting the presentation time corresponding toeach image in the image sequence based on a corresponding durationimpact parameter for each image in the image sequence includes: finding,for an image j, a presentation time offset of the image j from a thirdmapping table based on a duration impact parameter of the image j, wherethe third mapping table includes a plurality of duration impactparameters and presentation time offsets respectively corresponding tothe plurality of duration impact parameters; and adjusting apresentation time of the image j based on the presentation time offsetof the image j; where the image j is any image in the image sequence.

In an embodiment, the duration impact parameter includes the firstrecognition probability. The adjusting the presentation timecorresponding to each image in the image sequence based on acorresponding duration impact parameter for each image in the imagesequence includes: obtaining a presentation time offset of each image inthe image sequence by using the following fitting formula:

${{\Delta{T(c)}} = {\sum\limits_{t = 0}^{n}{a_{j}c^{t}}}};$where

ΔT(c) is the presentation time offset, c is the first recognitionprobability, c is a real number satisfying 0≤c≤1, n is an order at whichΔT(c) fits c, n is an integer greater than 0, t is an integer satisfying−n≤t≤n, and a_(t) is a coefficient of c^(t); and adjusting thepresentation time of each image in the image sequence based on thepresentation time offset of each image in the image sequence.

In an embodiment, ΔT(c) is obtained after n-order linear fitting isperformed on c by using (c1, T2) and (c2, T1). T1 is a minimumpresentation time threshold, T2 is a maximum presentation timethreshold, c1 is a minimum probability threshold of a recognitionprobability determined by using the computer vision algorithm, and c2 isa maximum probability threshold of a recognition probability determinedby using the computer vision algorithm.

In an embodiment, when the first recognition probability of the image qis greater than or equal to c2, the first recognition probability isused to determine that the image q includes the preset image feature.When the first recognition probability of the image q is less than orequal to c1, the first recognition probability is used to determine thatthe image q does not include the preset image feature. The image q isany image in the image sequence.

In an embodiment, the duration impact parameter includes the fatiguestate parameter. The adjusting the presentation time corresponding toeach image in the image sequence based on a corresponding durationimpact parameter for each image in the image sequence includes:obtaining a presentation time offset of each image in the image sequenceby using the following fitting formula:

${{\Delta{T(f)}} = {\sum\limits_{k = 0}^{m}{a_{k}f^{k}}}};$whereΔT(f) is the presentation time offset, f is the fatigue state parameter,m is an order at which ΔT(f) fits f, m is a positive integer greaterthan 0, k is an integer satisfying −m≤k≤m, and a_(k) is a coefficient off^(k); and adjusting the presentation time of each image in the imagesequence based on the presentation time offset of each image in theimage sequence.

In an embodiment, the duration impact parameter includes the firstrecognition probability and the fatigue state parameter. The adjustingthe presentation time corresponding to each image in the image sequencebased on a corresponding duration impact parameter for each image in theimage sequence includes: obtaining a presentation time offset of eachimage in the image sequence by using the following fitting formula:

${{\Delta{T\left( {c,f} \right)}} = {\sum\limits_{t = 0}^{n}{\sum\limits_{k = 0}^{m}{a_{t,k}c^{t}f^{k}}}}};$where ΔT(c, f) is the presentation time offset, c is the firstrecognition probability, f is the fatigue state parameter, m is an orderat which ΔT(c, f) fits f, n is an order at which ΔT(c, f) fits c, both nand m are positive integers greater than 0, t is an integer satisfying−n≤t≤n, k is an integer satisfying −m≤k≤m, c is a real number satisfying0≤c≤1, and a_(t,k) is a coefficient of c^(t)f^(k); and adjusting thepresentation time corresponding to each image in the image sequencebased on the corresponding presentation time offset of each image in theimage sequence.

In an embodiment, f is a normalized value, and f is a value between [0,1].

In an embodiment, the method further includes: for an image q, obtaininga fatigue state parameter of the image q based on fatigue stateinformation that is sent by a sensor and that is obtained when theobservation object watches the image p, where the image q is any imagein the image sequence other than the first image, and the image p is aprevious image of the image q.

In an embodiment, a fatigue state parameter of the first image is presetto 0.

In an embodiment, when it is detected that a corresponding fatigue stateparameter obtained when the observation object observes the image r isgreater than or equal to a first fatigue threshold, images to bedisplayed after the image r in the image sequence are controlled not tobe displayed, and an image whose first recognition probability isgreater than or equal to a first probability threshold in the images tobe displayed after the image r is obtained. When it is detected that thefatigue state parameter of the observation object is less than or equalto a second fatigue threshold, the image whose first recognitionprobability is greater than or equal to the first probability thresholdin the images to be displayed after the image r is controlled to besequentially displayed. When the fatigue state parameter of theobservation object is greater than or equal to the first fatiguethreshold, that is, when it is detected that the observation object isalready fatigued, image display is suspended, so that the observationobject rests, and images with a relatively high first recognitionprobability are selected in this period. When the observation object hasfinished rest, brain-computer combination image recognition is performedon these images again. After the foregoing process, image recognitionefficiency can be improved.

In an embodiment, there are at least two observation objects, and thefatigue state parameter is at least two fatigue state parametersrespectively generated when the at least two observation objects observea same image. A presentation time of an image u is positively correlatedwith a weighted sum of the at least two fatigue state parameters, wherethe image u is any image in the image sequence. A plurality ofobservation objects simultaneously perform brain-computer combinationimage recognition on an image in the image sequence. This can reduce arandom error caused by a subjective reason of an observation object in acase of one observation object, thereby improving accuracy ofbrain-computer combination image recognition.

In an embodiment, the fatigue state parameter includes at least twofatigue state parameters respectively generated when the at least twoobservation objects observe a same image. Fatigue state parameters of aplurality of observation objects are measured and an image presentationtime is determined by using the plurality of fatigue state parameters.In this way, a random error caused by a subjective reason of anobservation object in a case of one observation object can be reduced,and determining of the image presentation time is more accurate.Therefore, the accuracy of brain-computer combination image recognitioncan be improved.

In an embodiment, the fusing, for each image in the image sequence, acorresponding computer vision signal and a corresponding feedback signalto obtain a target recognition signal of each image in the imagesequence includes: determining, for each image in the image sequencebased on at least one of the first recognition probability, the fatiguestate parameter, and the presentation time, a first weight correspondingto each image in the image sequence, where the first weight is a weightused when the corresponding feedback signal is used to determine thetarget recognition signal, the first weight is inversely correlated withthe first recognition probability, the first weight is inverselycorrelated with the fatigue state parameter, and the first weight ispositively correlated with the presentation time; and fusing, for eachimage in the image sequence based on a corresponding first weight, acorresponding computer vision signal and a corresponding feedback signalto obtain the target recognition signal of each image in the imagesequence. When recognition accuracy of the computer vision algorithm ishigher, a fusion weight used by recognition by using the computer visionalgorithm may be increased, and a fusion weight used by brainrecognition of the observation object may be reduced, thereby reducing amiss detection rate. When the fatigue state parameter of the observationobject is larger, the weight of brain recognition of the observationobject may be reduced, and therefore the miss detection rate can bereduced. When an image presentation time is longer, a time forobservation by the observation object is longer, and therefore accuracyof brain recognition of the observation object is higher. In this case,the weight of brain recognition of the observation object may beincreased, so as to reduce a miss detection rate.

In an embodiment, the computer vision signal is a first recognitionprobability determined by using the computer vision algorithm. Beforethe fusing, for each image in the image sequence, a correspondingcomputer vision signal and a corresponding feedback signal to obtain atarget recognition signal of each image in the image sequence, themethod further includes: calculating, for each image in the imagesequence, a second recognition probability of each image in the imagesequence based on a corresponding feedback signal, where the secondrecognition probability is used to indicate a probability that theobservation object determines that the image includes the preset imagefeature; and the fusing, for each image in the image sequence, acorresponding computer vision signal and a corresponding feedback signalto obtain a target recognition signal of each image in the imagesequence includes: calculating, for each image in the image sequence, atarget recognition probability of each image in the image sequence basedon the corresponding first recognition probability and the correspondingsecond recognition probability.

In an embodiment, the computer vision signal is an image featuredetermined by using the computer vision algorithm. Before the fusing,for each image in the image sequence, a corresponding computer visionsignal and a corresponding feedback signal to obtain a targetrecognition signal of each image in the image sequence, the methodfurther includes: determining, for each image in the image sequencebased on a corresponding feedback signal, a feedback signal featurecorresponding to each image in the image sequence; and the fusing, foreach image in the image sequence, a corresponding computer vision signaland a corresponding feedback signal to obtain a target recognitionsignal of each image in the image sequence includes: performing, foreach image in the image sequence, feature fusion on the correspondingimage feature and the corresponding feedback signal feature, to obtain afused feature corresponding to each image in the image sequence; anddetermining, for each image in the image sequence, a target recognitionprobability of each image in the image sequence based on thecorresponding fused feature.

In an embodiment, S images are determined, from the image sequence basedon the target recognition probability of each image in the imagesequence, as images including the preset image feature, where the targetrecognition probabilities of the S images meet a preset condition, and Sis an integer less than or equal to N. The preset condition may be thatthe target recognition probability is greater than or equal to athreshold, or the preset condition may be that the S images are thefirst S images sorted in descending order according to the targetrecognition probabilities when target recognition probabilities ofimages in the image sequence are sorted in descending order.

In an embodiment, images whose corresponding target recognitionprobabilities are between a second probability threshold and a thirdprobability threshold in the image sequence are used as a new imagesequence. The new image sequence may be used to re-execute the methoddescribed in any one of the first aspect and the embodiments of thefirst aspect. Images, in the image sequence, with relatively greatuncertainty of whether the preset image feature is included are re-used,for a plurality of times, as a new image sequence for brain computercombination image recognition. In this way, a suspicious object in theimage sequence can be filtered out, a probability of misjudgment of theimage recognition device is reduced, and accuracy of brain computercombination image recognition can be improved.

In an embodiment, when a target recognition probability of any image inthe image sequence is less than or equal to the second probabilitythreshold, the image is not an image that includes the preset imagefeature. When a target recognition probability of any image in the imagesequence is greater than or equal to the third probability threshold,the image is an image that includes the preset image feature. The secondprobability threshold is less than or equal to the third probabilitythreshold.

In an embodiment, the feedback signal is an electroencephalogram signal.

According to a fourth aspect, an embodiment of this application providesan image recognition method, including: setting a presentation time of atarget image based on a duration impact parameter of the target image,where the presentation time of the target image is used to indicate atime period from a presentation start moment of the target image to apresentation start moment of a next adjacent image, the duration impactparameter includes at least one of a first recognition probability and afatigue state parameter, the first recognition probability is used toindicate a probability, obtained by using a computer vision algorithm,that an image includes a preset image feature, the fatigue stateparameter is used to indicate a fatigue degree of an observation objectwhen the observation object observes an image, the presentation time isinversely correlated with the first recognition probability, and thepresentation time is positively correlated with the fatigue stateparameter; obtaining a feedback signal generated when the observationobject observes the target image within the presentation time of thetarget image; and determining a target recognition probability of thetarget image based on a computer vision signal and the feedback signalof the target image, where the computer vision signal is the firstrecognition probability or an image feature that is of the target imageand that is determined by using the computer vision algorithm. A higherprobability that the computer vision algorithm recognizes that an imageincludes the preset image feature indicates that an image recognitiondevice may set a longer brain recognition duration. More attentionresources in a time dimension of the observation object are allocated toan image with greater uncertainty. This can reduce a miss detection rateof image recognition and improve efficiency of brain-computercollaboration image recognition. In addition, a larger fatigue stateparameter of the observation object indicates that the observationobject is more fatigued, and the observation object needs a relativelylong observation time to recognize whether an image includes the presetimage feature. Therefore, a relatively long presentation time of theimage is set, and an image presentation time is set more properly.Determining the presentation time of the image based on a fatigue degreeof the observation object can reduce miss detection cases caused bybrain fatigue of the observation object, thereby reducing the missdetection rate.

In an embodiment, before the setting a presentation time of a targetimage based on a duration impact parameter of the target image, themethod further includes: receiving M images from a camera device, whereM is an integer greater than 1; and selecting N images from the M imagesas the image sequence, where N is less than or equal to M. The targetimage is any image in the image sequence.

In an embodiment, before the obtaining a feedback signal generated whenthe observation object observes the target image within the presentationtime of the target image, the method further includes: obtaining, basedon the presentation time of each image in the image sequence, apresentation time sequence corresponding to the image sequence; andcontrolling, based on the presentation time sequence, to sequentiallydisplay the image sequence.

In an embodiment, the duration impact parameter includes the firstrecognition probability. The setting a presentation time of a targetimage based on a duration impact parameter of the target image includes:setting the presentation time of the target image based on a firstrecognition probability of the target image. Before the obtaining afeedback signal generated when the observation object observes thetarget image within the presentation time of the target image, themethod further includes: in a process of controlling to sequentiallydisplay the image sequence based on the presentation time sequence,obtaining a fatigue state parameter corresponding to the target image;and adjusting, based on the fatigue state parameter corresponding to thetarget image, a presentation time of an image displayed after the targetimage in the image sequence. First, the presentation time sequencecorresponding to the image sequence is determined according to thecomputer vision algorithm. When the image sequence is displayed insequence according to the presentation time sequence, when the brain andthe computer are combined to recognize the image in the image sequence,a process of determining the presentation time sequence of the imagesequence does not need to be executed. This can reduce data load ofparallel processing performed by the image recognition device duringrecognition of the target image, thereby reducing a miss detection rateof brain-computer combination image recognition. Second, a fatigue stateparameter corresponding to the observation object when the observationobject observes an image in the image sequence may be detected in realtime by using a fatigue detection apparatus, and a presentation time ofthe image is adjusted based on the real-time fatigue state parameter.The fatigue state parameter can assist in correcting the presentationtime of the image, and can reduce miss detection cases caused by brainfatigue of the observation object, thereby reducing the miss detectionrate.

In an embodiment, before the obtaining a feedback signal generated whenthe observation object observes the target image within the presentationtime of the target image, the method further includes: displaying thetarget image based on the presentation time of the target image. Thefatigue state parameter of the observation object is detected in realtime, and the presentation time of the image is determined based on thefatigue state parameter detected in real time. In this way, the fatiguestate parameter of the observation object is more accurate, thedetermined presentation time is more proper, and a miss detection ratecan be reduced.

In an embodiment, the setting a presentation time of a target imagebased on a duration impact parameter of the target image includes:finding, from a first mapping table based on the duration impactparameter of the target image, a presentation time corresponding to theduration impact parameter of the target image, where the first mappingtable includes a plurality of duration impact parameters andpresentation times respectively corresponding to the plurality ofduration impact parameters; and setting the presentation time of thetarget image to the presentation time corresponding to the durationimpact parameter of the target image.

In an embodiment, the duration impact parameter includes the firstrecognition probability. The setting a presentation time of the targetimage based on a duration impact parameter of the target image includes:setting the presentation time of the target image by using the followingfitting formula:

${{T(c)} = {\sum\limits_{t = 0}^{n}{a_{j}c^{t}}}};$where

T(c) is the presentation time, c is the first recognition probability, cis a real number satisfying 0≤c≤1, n is an order at which T(c) fits c, nis an integer greater than 0, t is an integer satisfying −n≤t≤n, anda_(t) is a coefficient of c^(t).

In an embodiment, T(c) is obtained after n-order linear fitting isperformed on c by using (c1, T2) and (c2, T1). T1 is a minimumpresentation time threshold, T2 is a maximum presentation timethreshold, c1 is a minimum probability threshold of a recognitionprobability determined by using the computer vision algorithm, and c2 isa maximum probability threshold of a recognition probability determinedby using the computer vision algorithm.

In an embodiment, the duration impact parameter includes the fatiguestate parameter. The setting a presentation time of the target imagebased on a duration impact parameter of the target image includes:setting the presentation time of the target image by using the followingfitting formula:

${{T(f)} = {\sum\limits_{k = 0}^{m}{a_{k}f^{k}}}};$where

T(f) is the presentation time, f is the fatigue state parameter, m is anorder at which T(f) fits f, m is a positive integer greater than 0, k isan integer satisfying −m≤k≤m, and a_(k) is a coefficient of f^(k).

In an embodiment, the duration impact parameter includes the firstrecognition probability and the fatigue state parameter. The setting apresentation time of the target image based on a duration impactparameter of the target image includes: setting the presentation time ofthe target image by using the following fitting formula:

${{T\left( {c,f} \right)} = {\sum\limits_{t = 0}^{n}{\sum\limits_{k = 0}^{m}{a_{t,k}c^{t}f^{k}}}}};$where

T(c, f) is the presentation time, c is the first recognitionprobability, f is the fatigue state parameter, m is an order at whichT(c, f) fits f, n is an order at which T(c, f) fits c, both n and m arepositive integers greater than 0, t is an integer satisfying −n≤t≤n, kis an integer satisfying −m≤k≤m, c is a real number satisfying 0≤c≤1,and a_(t,k) is a coefficient of c^(t)f^(k).

In an embodiment, the method further includes: when it is detected thata corresponding fatigue state parameter obtained when the observationobject observes the target image is greater than or equal to a firstfatigue threshold, controlling to stop displaying images to be displayedafter the target image in the image sequence, and obtaining an imagewhose first recognition probability is greater than or equal to a firstprobability threshold in the images to be displayed after the targetimage; and when it is detected that the fatigue state parameter of theobservation object is less than or equal to a second fatigue threshold,controlling to sequentially display the image whose first recognitionprobability is greater than or equal to the first probability thresholdin the images to be displayed after the target image. When the fatiguestate parameter of the observation object is greater than or equal tothe first fatigue threshold, that is, when it is detected that theobservation object is already fatigued, image display is suspended, sothat the observation object rests, and images with a relatively highfirst recognition probability are selected in this period. When theobservation object has finished rest, brain-computer combination imagerecognition is performed on these images again. According to theforegoing process, a miss detection rate of brain-computer combinationimage recognition can be reduced.

In an embodiment, there are at least two observation objects, and thefeedback signal is at least two feedback signals respectively generatedwhen the at least two observation objects observe a same image. Thedetermining a target recognition probability of the target image basedon a computer vision signal and the feedback signal of the target imageincludes: determining the target recognition probability of the targetimage based on the computer vision signal and the at least two feedbacksignals of the target image. A plurality of observation objectssimultaneously perform brain-computer combination image recognition onan image in the image sequence. This can reduce a random error caused bya subjective reason of an observation object in a case of oneobservation object, thereby improving accuracy of brain-computercombination image recognition.

In an embodiment, the fatigue state parameter is at least two fatiguestate parameters respectively generated when the at least twoobservation objects observe a same image. A presentation time of thetarget image is positively correlated with a weighted sum of the atleast two fatigue state parameters. Fatigue state parameters of aplurality of observation objects are measured and an image presentationtime is determined by using the plurality of fatigue state parameters.In this way, a random error caused by a subjective reason of anobservation object in a case of one observation object can be reduced,and determining of the image presentation time is more accurate.Therefore, the accuracy of brain-computer combination image recognitioncan be improved.

In an embodiment, the determining a target recognition probability ofthe target image based on a computer vision signal and the feedbacksignal of the target image includes: determining a first weight based onat least one of the first recognition probability of the target image,the fatigue state parameter corresponding to the target image, and thepresentation time of the target image, where the first weight is aweight used when the feedback signal of the target image is used todetermine the target recognition probability, the first weight isinversely correlated with the first recognition probability of thetarget image, the first weight is inversely correlated with the fatiguestate parameter of the target image, and the first weight is positivelycorrelated with the presentation time of the target image; and fusing,based on the first weight of the target image, the computer visionsignal of the target image and the feedback signal of the target imageto obtain the target recognition signal of each image in the imagesequence. When recognition accuracy of the computer vision algorithm ishigher, a fusion weight used by recognition by using the computer visionalgorithm may be increased, and a fusion weight used by brainrecognition of the observation object may be reduced, thereby reducing amiss detection rate. When the fatigue state parameter of the observationobject is larger, the weight of brain recognition of the observationobject may be reduced, and therefore the miss detection rate can bereduced. When an image presentation time is longer, a time forobservation by the observation object is longer, and therefore accuracyof brain recognition of the observation object is higher. In this case,the weight of brain recognition of the observation object may beincreased, so as to reduce a miss detection rate.

In an embodiment, the computer vision signal is the first recognitionprobability. Before the determining a target recognition probability ofthe target image based on a computer vision signal and the feedbacksignal of the target image, the method further includes: calculating asecond recognition probability of the target image based on the feedbacksignal of the target image, where the second recognition probability isused to indicate a probability that the observation object determinesthat the target image includes the preset image feature. The determininga target recognition probability of the target image based on a computervision signal and the feedback signal of the target image includes:calculating the target recognition probability of the target image basedon the first recognition probability of the target image and the secondrecognition probability of the target image.

In an embodiment, the computer vision signal is an image feature that isof the target image and that is determined by using the computer visionalgorithm. Before the determining a target recognition probability ofthe target image based on a computer vision signal and the feedbacksignal of the target image, the method further includes: determining,based on the feedback signal of the target image, a feedback signalfeature of the observation object when the observation object observesthe target image; and performing feature fusion on the image feature ofthe target image and the feedback signal feature of the observationobject when the observation object observes the target image, to obtaina mixed feature of the target image. The determining a targetrecognition probability of the target image based on a computer visionsignal and the feedback signal includes: determining the targetrecognition probability of the target image based on the mixed featureof the target image.

In an embodiment, the method further includes: determining, from theimage sequence based on the target recognition probability of each imagein the image sequence, S images as images including the preset imagefeature, where the target recognition probabilities of the S images meeta preset condition, and S is an integer less than or equal to N. Thepreset condition may be that the target recognition probability isgreater than or equal to a threshold, or the preset condition may bethat the S images are the first S images sorted in descending orderaccording to the target recognition probabilities when targetrecognition probabilities of images in the image sequence are sorted indescending order.

In an embodiment, the method further includes: using images whosecorresponding target recognition probabilities are between a secondprobability threshold and a third probability threshold in the imagesequence as a new image sequence. Images, in the image sequence, withrelatively great uncertainty of whether the target image feature isincluded are selected, for a plurality of times, as a new image sequencefor brain-computer combination image recognition. In this way, asuspicious object in the image sequence can be filtered out, aprobability of misjudgment by the image recognition device is reduced,and accuracy of brain-computer combination image recognition can beimproved.

In an embodiment, the feedback signal is an electroencephalogram signal.

According to a fifth aspect, an embodiment of this application providesan image recognition device, including a processor and a memory, wherethe memory is configured to store a program instruction, and theprocessor is configured to invoke the program instruction to execute thebrain-computer combination image recognition method based on imagesequence presentation provided in any one of the first aspect and theembodiments of the first aspect.

According to a sixth aspect, an embodiment of this application providesan image presentation time determining device, including a processor anda memory, where the memory is configured to store a program instruction,and the processor is configured to invoke the program instruction toexecute the image presentation time determining method provided in anyone of the second aspect and the embodiments of the second aspect.

According to a seventh aspect, an embodiment of this applicationprovides an image presentation time adjustment device, including aprocessor and a memory, where the memory is configured to store aprogram instruction, and the processor is configured to invoke theprogram instruction to execute the image presentation time adjustmentmethod provided in any one of the third aspect and the embodiments ofthe third aspect.

According to an eighth aspect, an embodiment of this applicationprovides an image recognition device, including a processor and amemory, where the memory is configured to store a program instruction,and the processor is configured to invoke the program instruction toexecute the image recognition method provided in any one of the fourthaspect and the embodiments of the fourth aspect.

According to a ninth aspect, an embodiment of this application providesan image recognition device, where the device includes a module or aunit that is configured to execute the brain-computer combination imagerecognition method based on image sequence presentation provided in anyone of the first aspect and the embodiments of the first aspect.

According to a tenth aspect, an embodiment of this application providesan image presentation time determining device, where the device includesa module or a unit that is configured to execute the image presentationtime determining method provided in any one of the second aspect and theembodiments of the second aspect.

According to an eleventh aspect, an embodiment of this applicationprovides an image presentation time adjustment device, where the deviceincludes a module or a unit that is configured to execute the imagepresentation time adjustment method provided in any one of the thirdaspect and the embodiments of the third aspect.

According to a twelfth aspect, an embodiment of this applicationprovides an image recognition device, where the device includes a moduleor a unit that is configured to execute the image recognition methodprovided in any one of the fourth aspect and the embodiments of thefourth aspect.

According to a thirteenth aspect, an embodiment of the present inventionprovides a chip system, where the chip system includes at least oneprocessor, a memory, and an interface circuit. The memory, the interfacecircuit, and the at least one processor are interconnected by using aline, and the at least one memory stores a program instruction. When theprogram instruction is executed by the processor, the method describedin any one of the first aspect and the embodiments of the first aspectis implemented.

According to a fourteenth aspect, an embodiment of the present inventionprovides a chip system, where the chip system includes at least oneprocessor, a memory, and an interface circuit. The memory, the interfacecircuit, and the at least one processor are interconnected by using aline, and the at least one memory stores a program instruction. When theprogram instruction is executed by the processor, the method describedin any one of the second aspect and the embodiments of the second aspectis implemented.

According to a fifteenth aspect, an embodiment of the present inventionprovides a chip system, where the chip system includes at least oneprocessor, a memory, and an interface circuit. The memory, the interfacecircuit, and the at least one processor are interconnected by using aline, and the at least one memory stores a program instruction. When theprogram instruction is executed by the processor, the method describedin any one of the third aspect and the embodiments of the third aspectis implemented.

According to a sixteenth aspect, an embodiment of the present inventionprovides a chip system, where the chip system includes at least oneprocessor, a memory, and an interface circuit. The memory, the interfacecircuit, and the at least one processor are interconnected by using aline, and the at least one memory stores a program instruction. When theprogram instruction is executed by the processor, the method describedin any one of the fourth aspect and the embodiments of the fourth aspectis implemented.

According to a seventeenth aspect, an embodiment of the presentinvention provides a computer readable storage medium, where thecomputer readable storage medium stores a program instruction, and whenthe program instruction is run by a processor, the method described inany one of the first aspect and the embodiments of the first aspect isimplemented.

According to an eighteenth aspect, an embodiment of the presentinvention provides a computer readable storage medium, where thecomputer readable storage medium stores a program instruction, and whenthe program instruction is run by a processor, the method described inany one of the second aspect and the embodiments of the second aspect isimplemented.

According to a nineteenth aspect, an embodiment of the present inventionprovides a computer readable storage medium, where the computer readablestorage medium stores a program instruction, and when the programinstruction is run by a processor, the method described in any one ofthe third aspect and the embodiments of the third aspect is implemented.

According to a twentieth aspect, an embodiment of the present inventionprovides a computer readable storage medium, where the computer readablestorage medium stores a program instruction, and when the programinstruction is run by a processor, the method described in any one ofthe fourth aspect and the embodiments of the fourth aspect isimplemented.

According to a twenty-first aspect, an embodiment of the presentinvention provides a computer program product, and when the computerprogram product is run by a processor, the method described in any oneof the first aspect and the embodiments of the first aspect isimplemented.

According to a twenty-second aspect, an embodiment of the presentinvention provides a computer program product, and when the computerprogram product is run by a processor, the method described in any oneof the second aspect and the embodiments of the second aspect isimplemented.

According to a twenty-third aspect, an embodiment of the presentinvention provides a computer program product, and when the computerprogram product is run by a processor, the method described in any oneof the third aspect and the embodiments of the third aspect isimplemented.

According to a twenty-fourth aspect, an embodiment of the presentinvention provides a computer program product, and when the computerprogram product is run by a processor, the method described in any oneof the fourth aspect and the embodiments of the fourth aspect isimplemented.

According to a twenty-fifth aspect, an embodiment of this applicationprovides an image recognition system, including: an image recognitiondevice, a display device, and a feedback signal collection device, wherethe image recognition device is separately connected to the displaydevice and the feedback signal collection device. The image recognitiondevice is configured to execute the brain-computer combination imagerecognition method based on image sequence presentation provided in anyone of the first aspect and the embodiments of the first aspect. Thedisplay device is configured to display the image sequence. The feedbacksignal collection device is configured to: obtain a feedback signalobtained when the observation object observes any image i in the imagesequence, and feed back the feedback signal to the image recognitiondevice.

Specifically, the image recognition device may be the image recognitiondevice described in the fifth aspect or the ninth aspect.

According to a twenty-sixth aspect, an embodiment of this applicationprovides an image recognition system, including: an image presentationtime determining device, a display device, and a feedback signalcollection device, where the image presentation time determining deviceis separately connected to the display device and the feedback signalcollection device. The image presentation time determining device isconfigured to execute the image presentation time determining methodprovided in any one of the second aspect and the embodiments of thesecond aspect. The display device is configured to display the imagesequence. The feedback signal collection device is configured to: obtaina feedback signal obtained when the observation object observes anyimage i in the image sequence, and feed back the feedback signal to theimage presentation time determining device.

Specifically, the image presentation time determining device may be theimage presentation time determining device described in the sixth aspector the tenth aspect.

According to a twenty-seventh aspect, an embodiment of this applicationprovides an image recognition system, including: an image presentationtime adjustment device, a display device, and a feedback signalcollection device, where the image presentation time adjustment deviceis separately connected to the display device and the feedback signalcollection device. The image presentation time adjustment device isconfigured to execute the image presentation time adjustment methodprovided in any one of the third aspect and the embodiments of the thirdaspect. The display device is configured to display the image sequence.The feedback signal collection device is configured to: obtain afeedback signal obtained when the observation object observes any imagei in the image sequence, and feed back the feedback signal to the imagepresentation time adjustment device.

Specifically, the image presentation time adjustment device may be theimage presentation time determining device described in the seventhaspect or the eleventh aspect.

According to a twenty-eighth aspect, an embodiment of this applicationprovides an image recognition system, including: an image recognitiondevice, a display device, and a feedback signal collection device, wherethe image recognition device is separately connected to the displaydevice and the feedback signal collection device. The image recognitiondevice is configured to execute the image recognition method provided inany one of the fourth aspect and the embodiments of the fourth aspect.The display device is configured to display the target image. Thefeedback signal collection device is configured to: obtain a feedbacksignal obtained when the observation object observes the target image,and feed back the feedback signal to the image recognition device.

Specifically, the image recognition device may be the image recognitiondevice described in the eighth aspect or the twelfth aspect.

DESCRIPTION OF DRAWINGS

The following describes the accompanying drawings used in theembodiments of this application.

FIG. 1 is a schematic diagram of an architecture of an image recognitionsystem according to an embodiment of this application;

FIG. 2 is a schematic structural diagram of an electroencephalogramcollection device according to an embodiment of this application;

FIG. 3 is a schematic structural diagram of an image recognition deviceaccording to an embodiment of this application;

FIG. 4 is a schematic flowchart of a brain-computer combination imagerecognition method based on image sequence presentation according to anembodiment of this application;

FIG. 5 is a schematic diagram of an image presentation time according toan embodiment of this application;

FIG. 6 is a schematic flowchart of an image presentation time adjustmentmethod according to an embodiment of this application;

FIG. 7 is a schematic structural diagram of an image recognition deviceaccording to an embodiment of this application;

FIG. 8 is a schematic structural diagram of an image presentation timedetermining device according to an embodiment of this application;

FIG. 9 is a schematic structural diagram of an image presentation timeadjustment device according to an embodiment of this application; and

FIG. 10 is a schematic structural diagram of an image recognition deviceaccording to an embodiment of this application.

DESCRIPTION OF EMBODIMENTS

The embodiments of the present invention are described below withreference to the accompanying drawings in the embodiments of the presentinvention. Terms used in an implementation part of this application aremerely used to explain specific embodiments of this application, and arenot intended to limit this application.

To help understanding of the embodiments of this application, thefollowing describes some terms used in the embodiments of thisapplication.

(1) Rapid Serial Visual Presentation Paradigm

Rapid serial visual representation is an experimental paradigm based onvisual image sequence stimulation. In a process of executing a rapidserial visual representation paradigm task, a sequence of images thatare displayed in chronological order at intervals may be presented to anobservation object. In a process of displaying the image sequence, eachimage including a stimulus may be presented at a same location on adisplay device, a next image appears after a previous image disappears,and presentation times of all the images may be equal. The imagesequence presented to the observation object may be used as an externalstimulus to a brain of the observation object, so that the observationobject generates a feedback signal. Feedback signals generated when theobservation object observes the image sequence may be collected by usinga device. A specific feedback signal is generated when the observationobject observes an image of interest in the image sequence. Therefore, afeedback signal of the observation object may be detected, and whetherthe feedback signal complies with the specific feedback signal or asignal feature of the specific feedback signal is detected, to obtain,through reverse analysis, an image with a preset image feature includedin the image sequence. The observation object may be, for example, aperson or an animal. For any image i in the image sequence, a feedbacksignal may be analyzed to learn whether the image includes the presetimage feature. The target image described below may also be any image inthe image sequence.

(2) Feedback Signal

A feedback signal is used to indicate a reaction of the observationobject to a watched image. When the observation object observes an imagesequence based on RSVP, one feedback signal is generated for each imagein the image sequence, and the feedback signal may be a biologicalsignal of the observation object collected by using a sensor. Thefeedback signal in this application may be any one or more of thefollowing: an electroencephalogram signal, an eye movement signal, askin potential reaction signal, a body movement reaction signal, and thelike. This application is described by using an electroencephalogramsignal as an example. It may be understood that the feedback signal inthis application may be not limited to the foregoing several feedbacksignals. With progress of technologies, a detected feedback signal of anorganism may also be a biological signal that can be detected and thatnewly appears in the future.

The electroencephalogram signal is a biological electrical signal, andmay include a spontaneous electroencephalogram signal and an evokedpotential (EP) electroencephalogram signal. The spontaneouselectroencephalogram signal is a spontaneous change in electricalpotential of nerve cells in the brain without specific externalstimulation. The evoked potential electroencephalogram signal is achange of brain potential caused by external stimulation of neurons inthe brain, such as sound, light, electricity, or a specific image. Theelectroencephalogram signal in this application is an inductiveelectroencephalogram signal. When an observation object observes animage sequence, a group of continuous electroencephalogram signals ofthe observation object may be continuously collected. Then detection isperformed on the group of electroencephalogram signals. When anelectroencephalogram signal including a first electroencephalogramsignal feature is detected, an image observed by the observation objectwhen the electroencephalogram signal including the firstelectroencephalogram signal feature is generated may be obtained, so asto implement image recognition.

An electroencephalogram signal can be collected by using abrain-computer interface (BCI). The brain-computer interface creates achannel for the brain of an observation object to communicate with anexternal device, and is used to collect electroencephalogram signals ofthe observation object, convert the electroencephalogram signals intodata that can be recognized by a computer, and decode a thinkingintention of the observation object through data analysis, so that imagerecognition can be implemented through brain-computer cooperation.

Event related potential (ERP) may be used to represent anelectroencephalogram signal. The event related potential indicatesvoltage fluctuation of an electroencephalogram signal over time, whichcan be evoked by visual or auditory stimulation. The evoked voltagefluctuation of the electroencephalogram signal can be called “peak”,“wave”, or “electroencephalogram component”. The voltage fluctuation ofthe electroencephalogram signal is a main object of electroencephalogramsignal research. A P300 component is a most commonly usedelectroencephalogram component for detecting whether a preset imagefeature is included.

In this application, an image that is recognized from the image sequenceand includes the preset image feature may be a specific object in theimage sequence. This application does not limit specific content andexpression form of the specific object. For example, the specific objectmay be a “dangerous object” in video detection in a security protectionscenario, for example, a “cutting tool or gun”. For another example, a“criminal suspect” is recognized from an image sequence that includes animage of the “criminal suspect”, and the image of the “criminal suspect”is a specific object. The specific object may not be limited to aspecific target, and there may be various types of specific objects, orspecific objects of one category or a plurality of categories, which isnot limited in this application. For example, a specific object may be a“cutting tool or gun”, or may be a target of a “dangerous object”category, or may be targets of a plurality of categories such as“dangerous objects” and “prohibited objects”. It may be understood thatthe foregoing example for a specific object should not constitute alimitation. As long as a small quantity of target stimuli are doped in alarge quantity of non-target stimuli, and when the target stimulusappears, a specific feedback signal is triggered, the small quantity oftarget stimuli can be understood as specific objects in thisapplication.

If the feedback signal is an electroencephalogram signal, when aspecific object appears, a particular feedback signal P300 component istriggered. P300 may be interpreted as about 300 ms or a longer timeafter presentation of a specific object, for example, may be a positivepeak that appears within up to 900 ms. When a specific object ispresented, the brain of the observation object may generate a P300component. When a non-specific object is presented, the brain of theobserved object does not generate a P300 component. Based on the aboveprinciple, ERP can be used to recognize a specific objects in an imagesequence.

When it is detected that an electroencephalogram signal meets a specificelectroencephalogram signal feature, it may be reversely determined thatan image observed by an observation object when the electroencephalogramsignal is generated is a specific object. The specificelectroencephalogram signal feature is an electroencephalogram featuremet by an electroencephalogram signal generated by the observationobject when the observation object observes the specific object.Specifically, the specific electroencephalogram signal feature is apositive peak that appears about 300 ms after the specific object isobserved. In other words, the specific electroencephalogram signalfeature is an electroencephalogram feature of P300. When it is detectedthat a positive peak appears in the electroencephalogram signal, it maybe determined that in the image sequence, an image observed by theobservation object about 300 ms before the positive peak appears is animage that includes a preset image feature; alternatively, a probabilitythat an image observed by the observation object about 300 ms before thepositive peak appears is an image that includes a preset image featureis determined. An image recognition device may also calculate, based ona feature of an electroencephalogram signal and a computer vision signalobtained by using a computer vision algorithm, a probability that animage includes the preset image feature (which may be referred to as atarget recognition probability below). The image recognition devicedetermines, based on the target recognition probability, whether theimage for generating the target recognition probability is an image thatincludes the preset image feature.

The foregoing describes a principle of brain recognition of anobservation object by using a feedback signal as an electroencephalogramsignal. The feedback signal is not limited to an electroencephalogramsignal, and may be extended to one or more of an electroencephalogramsignal, an eye movement signal, a skin potential reaction signal, and abody movement reaction signal, and may also be extended to a newbiological signal that can be detected in the future.

(3) Computer Vision Algorithm

The computer vision algorithm is a mathematical model used for helping acomputer understand an image. A core idea of the computer visionalgorithm is to use a data-driven method to learn a statistics featureand pattern from big data. Generally, a large quantity of trainingsamples are required to train the model. Specifically, the computervision algorithm may be used to model image features including texture,color, shape, spatial relationship, high-level semantics, and the like.An initial model is trained by using a training sample, and parametersin the initial model are adjusted to converge an error of imagerecognition, so as to construct a new model. After the training iscompleted, a probability that an image in the image sequence is aspecific object may be predicted by using the new model, so as toperform image recognition.

Currently, the computer vision algorithm may include a conventionalimage detection algorithm and a deep learning algorithm based on anartificial neural network. A process of performing image recognition byusing a conventional image recognition algorithm is as follows: First, afeature extraction algorithm is used to extract an image feature. Theimage feature may be one or more of a color feature, a texture feature,a shape feature, a spatial relationship feature, or a high-levelsemantic feature of the image. A method for extracting an image featuremay be implemented by using one or more of the following algorithms: forexample, a local binary pattern (LBP) algorithm, a histogram of orientedgradient (HOG) algorithm, or a Haar feature algorithm. Then, theextracted image feature may be input into a support vector machine (SVM)to calculate a probability that the image includes the preset imagefeature.

When a conventional image recognition algorithm is used to perform imagerecognition, another feature extraction algorithm may also be used, forexample, a deformable parts model (DPM) algorithm. This is not limitedin this application. A process of calculating a probability by using aconventional image recognition algorithm may further include anotheroperation on an image, for example, removing an image background throughpreprocessing, and segmenting the image. This is not limited in thisapplication.

Another most commonly used computer vision algorithm is the deeplearning algorithm based on an artificial neural network. In the deeplearning algorithm based on an artificial neural network, an imagefeature may be extracted by using a plurality of neural network layers,and a probability that the image includes a preset image feature iscalculated. The deep learning algorithm based on an artificial neuralnetwork may be, for example, a convolutional neural network (CNN). Thedeep learning algorithm may use the convolutional neural network toextract an image feature and calculate a probability that the imageincludes a preset image feature. The convolutional neural network usedfor image recognition may be considered as a classifier, and images areclassified by using the convolutional neural network, and may beclassified into an image that includes the preset image feature and animage that does not include the preset image feature. The convolutionalneural network may be a new model obtained after parameters in aninitial model of a specific network architecture are adjusted by using atraining sample to converge a recognition error. The parameters in themodel may include a convolution core size, a pooling core size, aquantity of fully connected layers, and the like.

An operation of performing image recognition by using the trainedconvolutional neural network model is: First, an image is processed byusing a convolutional layer, so as to extract information that is in theimage and that carries an image feature, where a specific form of theinformation may be a sequence; second, a data amount of the imagefeature information is reduced by using a pooling layer; and finally,information that carries the image feature and that is output by thepooling layer is input to a fully connected layer, to determine whetherthe image includes the preset image feature. In other words, whether theimage includes the preset image feature is predicted. The lastconvolutional layer may also be a classifier, for example, a softmaxmodel or an SVM, which predicts, based on input information that carriesthe image feature, whether the image includes the preset image feature.

It should be noted that the operations of performing image recognitionby using the foregoing convolutional neural network may be increased ordecreased. The foregoing operations may be combined and stacked for anyquantity of times to perform actual image recognition. The foregoingdeep learning algorithm based on an artificial neural network may alsobe an algorithm other than the convolutional neural network, which isnot limited in this application.

(4) The Brain of the Observation Object Works with the Computer toPerform Image Recognition.

That the brain of the observation object works with the computer toperform image recognition means to combine a feedback signal obtainedafter a target image in the image sequence is recognized by theobservation object and a recognition result of a computer visionalgorithm, so as to perform image recognition. Specifically, a computervision signal of the target image may be obtained after the target imageis processed by using the computer vision algorithm. The computer visionsignal may be a probability that the target image includes a presetimage feature and that is obtained through calculation by using thecomputer vision algorithm, or may be an image feature of the targetimage that is obtained through calculation by using the computer visionalgorithm. After the feedback signal of the observation object isobtained, the probability that the target image includes the presetimage feature may be determined based on the feedback signal, or afeedback signal feature of the feedback signal may be extracted. Thetarget image herein may be understood as any image i in the imagesequence. When the image recognition device performs image recognitionby combining a computer vision signal and a feedback signal, twocombination methods may be used: a probability weighted sum method and afeature fusion method. Descriptions are separately provided in thefollowing:

a. Using a Probability Weighted Sum to Fuse a Computer Vision Signal anda Feedback Signal:

The image recognition device processes the target image by using thecomputer vision algorithm, to obtain a probability that the target imageincludes the preset image feature, namely a first recognitionprobability p1. Then, the image recognition device obtains a feedbacksignal generated when the observation object observes the target image,and calculates, based on the feedback signal, a probability p2 that thetarget image includes the preset image feature, which may be referred toas a second recognition probability. In this way, the image recognitiondevice may obtain a probability that the target image is an imageincluding the preset image feature, that is, a target recognitionprobability p of the target image:p=w1*p1+w2*p2  (1-1)

Herein, w1 is a weight of the first recognition probability p1 obtainedthrough recognition by using the computer vision algorithm, and w2 is aweight of the second recognition probability p2 obtained throughrecognition by the observation object. Values of w1 and w2 may bedetermined according to experience, or may be learned by one or moreobservation objects. Generally, in image recognition performed by usingthe computer vision algorithm and image recognition performed by anobservation object, a higher recognition accuracy of the two indicates alarger weight. A sum of w1 and w2 may be normalized to 1.

The image recognition device may determine, based on the targetrecognition probability p of the target image, whether the target imageis an image that includes the preset image feature. For example, athreshold may be set. When the target recognition probability p of thetarget image is greater than or equal to the threshold, the imagerecognition device determines that the image is an image including thepreset image feature. The image recognition device may also sort animage sequence according to target recognition probabilities, which maybe sorting in descending order. The image recognition device selectsfirst N images in the sorted image sequence as images including thepreset image feature.

b. Feature Fusion for Brain-Computer Combination Image Recognition:

A feedback signal is generated when the observation object observes atarget image, feature fusion is performed on a feedback signal featureextracted from the feedback signal and an image feature extracted afterthe target image is processed by using a computer vision algorithm, anda target recognition probability is calculated based on a result of thefeature fusion.

Specifically, an example in which the feedback signal is anelectroencephalogram signal is used for description. A two-dimensionalmatrix may be obtained through short-time Fourier transform (STFT) of anelectroencephalogram signal with a single electrode. A transversedirection of the matrix is a time axis, and a longitudinal direction isa frequency component axis of the electroencephalogram signal. Thetwo-dimensional matrix is used to represent a tensor feature of anelectroencephalogram signal generated when an observation objectobserves a target image, and the tensor feature may be an energy featureor a power spectrum density feature of the electroencephalogram signal.A plurality of two-dimensional matrices may be obtained from anelectroencephalogram signal with a plurality of electrodes, and theplurality of two-dimensional matrices are spliced into athree-dimensional tensor in a third dimension. Three dimensions of thethree-dimensional tensor are respectively a time, a frequency of theelectroencephalogram signal, and a channel (a quantity of channels isthe same as a quantity of electrodes for collecting theelectroencephalogram signal).

The image recognition device may extract an image feature from theforegoing target image by using a convolutional neural network. Theimage recognition device obtains, through extraction, a two-dimensionalfeature map by using one convolutional layer or a plurality ofconvolutional layers, where the feature map is an image featurecorresponding to all pixels in a width direction and a height directionof a two-dimensional target image. The convolutional layer may be aplurality of layers, and the extracted image feature of the target imagemay also be a three-dimensional tensor feature. The three dimensions area width of the target image, a height of the target image, and theconvolutional layer respectively.

Feature fusion may be that the image recognition device performs tensorsplicing on a three-dimensional tensor obtained from theelectroencephalogram signal generated when the observation objectobserves the target image and a three-dimensional image feature of thetarget image that is obtained by using the convolutional neural network,so as to obtain a fused three-dimensional tensor. The fusedthree-dimensional tensor not only reflects a feature of theelectroencephalogram signal obtained through electroencephalogramrecognition on the target image, but also reflects an image featureobtained after image recognition is performed on the target image byusing the computer vision algorithm, and may be referred to as anelectroencephalogram visual hybrid feature.

After the electroencephalogram visual hybrid feature of the target imageis obtained, the electroencephalogram visual hybrid feature of thetarget image is processed by using a classifier layer in theconvolutional neural network, and the classifier layer is configured tooutput the target recognition probability of the target image based onthe electroencephalogram visual hybrid feature of the target image.

The performing image feature extraction and fusion to obtain theelectroencephalogram visual hybrid feature and outputting the targetrecognition probability by using the classifier may be performed in oneconvolutional neural network. The foregoing convolutional neural networkmay be obtained through training.

It may be understood that the foregoing two combination methods are onlyused to describe the embodiments of this application, and should notconstitute a limitation. The embodiments of this application do notlimit a combination method used for combining a computer visionalgorithm and an electroencephalogram signal to perform imagerecognition.

(5) First Recognition Probability Obtained Through Calculation by aComputer Vision Algorithm

A method for determining, by the image recognition device by using thecomputer vision algorithm, a probability that the target image includesthe preset image feature may be classifying by using a classifier. Thefollowing describes, by using an example in which a softmax classifieris used, obtaining of the probability that the target image includes thepreset image feature.

A classification target of the softmax classifier is to output aprobability that the target image in the image sequence is a specificobject and a probability that the target image is a non-specific object.An input of the softmax classifier may be an image feature of the targetimage, and an output is the probability p0 that the target image is aparticular object and the probability p0′ that the target image is anon-particular object. The two probability values may be respectivelyoutput by using two neurons. p0+p0′=1, p0 and p0′ are constants whosevalue ranges are [0, 1]. The probability p0 that the target image is aparticular object may be used as the first recognition probability ofthe target image that is obtained through calculation by using thecomputer vision algorithm, that is, c=p0.

In addition, the first recognition probability c of the target imagethat is obtained through calculation by using the computer visionalgorithm may be a function of p0, and is used to represent aprobability that the target image includes the preset image feature,that is, c=f(p0), where f(p0) is the probability that the target imageincludes the preset image feature. For example, f(p0) may be a maximumvalue function max(p0.1−p0), that is, calculating a maximum valuebetween p0 and p0′, and in this case, a value range of the firstrecognition probability c is [0.5, 1]. For another example, f(p0) mayalso be |p0−0.5|. A function relationship between the first recognitionprobability and p0 is not limited in this application. The firstrecognition probability may be used to reflect a reliability degree of aresult of image recognition performed by the classifier.

(6) Fatigue State Parameter

A fatigue state parameter of the observation object can be obtained bythe image recognition device by analyzing a feedback signal of theobservation object, or may be detected by a related sensor by measuringfatigue state information or a fatigue degree of the observation object,or may be obtained through prediction by using a fatigue rule.

For example, when the feedback signal is an electroencephalogram signal,an electroencephalogram collection device may be used as a sensor formeasuring fatigue status information. The image recognition device mayperform calculation for the electroencephalogram signal by using afatigue detection algorithm, to obtain the fatigue state parameter ofthe observation object. Specifically, a principle of the fatiguedetection algorithm is: When the observation subject is in a fatiguestate, β waves and high-frequency electroencephalogram of the braindecrease and α waves increase. A frequency of the electroencephalogramsignal gradually decreases to a slow wave (θ wave) when the observationobject changes from the fatigue state to a dozing state or sleep state.In the industry, an electroencephalogram signal power spectrum ratio:(α+θ)/β is used as the fatigue state parameter to describe a fatiguedegree of the observation object. The fatigue state parameter may alsobe obtained through normalization, and may be a value between [0, 1].

In addition to representing the fatigue state parameter of theobservation object by using an electroencephalogram signal, the fatiguestate parameter can also be detected based on an eye movement of theobservation object. In other words, fatigue status information isdetected by using a sensor for detecting an eye movement. A specificprinciple is as follows: A plurality of eye movement indicators may bedetected, for example, a blinking frequency, a time proportion of eyeclosure in a preset time, a gaze direction, and a gaze time; and the eyemovement indicators are calculated by using a fatigue state parameteralgorithm model to obtain the fatigue state parameter of the observationobject.

In addition to the foregoing real-time detection of the fatigue stateparameter of the observation object, a fatigue state parameter of theobservation object when the observation object observes any image in theimage sequence may be predicted according to a fatigue rule. The fatiguerule may be an objectively existing change rule of a fatigue degree ofthe observation object. The fatigue rule may be used to indicate achange rule of the fatigue degree of the observation object based on aquantity of images observed by the observation object, or the fatiguerule is used to indicate a change rule of the fatigue degree of theobservation object based on a duration spent by the observation objectfor image observation. In the embodiments of this application, thefatigue rule may be in a form of a mapping table, or may be in a form ofa fitting formula. For details, refer to a subsequent embodiment.

The fatigue rule may also be obtained through training by using aplurality of samples for one or more observation objects. Each sample inthe plurality of samples is a combination of a quantity of observedimages and a fatigue state parameter, or each sample in the plurality ofsamples is a combination of a duration spent for image observation and afatigue state parameter.

In the embodiments of this application, a method for representing afatigue state parameter is not limited to the foregoing method, and thefatigue state parameter may be detected and represented in anothermanner. This is not limited in this application.

FIG. 1 is a schematic diagram of an architecture of an image recognitionsystem according to an embodiment of this application. As shown in FIG.1, the image recognition system 100 includes a display device 10, afeedback signal collection device 20, an observation object 30, and animage recognition device 40. The image recognition device 40 isconnected to the feedback signal collection device 20, and is configuredto obtain a feedback signal of the observation object 30 collected bythe feedback signal collection device 20. The image recognition device40 is connected to the display device 10, and is configured to transmitan image sequence to the display device 10 for displaying for theobservation object 30 to watch.

The display device 10 is configured to: receive the image sequence sentby the image recognition device 40, and sequentially display the imagesequence in a specified presentation time sequence.

The feedback signal collection device 20 is configured to collect afeedback signal of the observation object 30, where the feedback signalmay be generated when the observation object 40 watches the imagesequence displayed on the display device 10. The feedback signalcollection device 20 may send the generated feedback signal to the imagerecognition device 40. The feedback signal collection device 20 may be,for example, a device including a brain-computer a interface used tocollect an electroencephalogram signal, and may be specifically, forexample, a head mounted electroencephalogram hat.

The observation object 30 may be a person, and specifically may be aspecific group of people, for example, a criminal investigator. Theobservation object 30 observes the image sequence displayed by thedisplay device 10, and generates a feedback signal to be collected bythe feedback signal collecting apparatus 20.

The image recognition device 40 is configured to calculate, by using acomputer vision algorithm pre-stored in the image recognition device 40,a probability that each image in the image sequence includes a presetimage feature. The image recognition device 40 is further configured toreceive the feedback signal collected by the feedback signal collectiondevice 20, where a feedback signal of the observation object collectedwhen the observation object observes a target image may be used torecognize the target image, so as to learn whether the target imageincludes the preset image feature or obtain a probability that thetarget image includes the preset image feature.

The brain has disadvantages of low efficiency and fatigability, but thebrain has abundant cognition and knowledge without training, and canalso recognize higher order semantic features. The computer visionalgorithm has disadvantages of a large error and difficult to extracthigher order semantic features, but the computer vision algorithm has anadvantage of high efficiency. Therefore, the image recognition device 40may be further configured to combine a recognition result of thecomputer vision algorithm and a recognition result of the brain of theobservation object 30, to perform image recognition. In other words, theimage recognition device 40 is further configured to fuse the feedbacksignal and a computer vision signal that is obtained through calculationby using the computer vision algorithm, to obtain a target recognitionprobability of each image in the image sequence through calculation. Therecognition result of the computer vision algorithm and the recognitionresult of the brain of the observation object 30 are combined, so thatadvantages of both brain recognition and computer algorithm recognitioncan be combined, and a miss detection rate of image recognition can beincreased. In addition, the image recognition device 40 may be furtherconfigured to obtain the image sequence. The image sequence may beextracted from a video stream. The video stream may be collected by acamera device such as a camera (not shown in FIG. 1).

It should be noted that the image recognition system 100 shown in FIG. 1is merely intended to more clearly describe the technical solutions inthis application, but is not intended to limit this application. Aperson of ordinary skill in the art may know that as a systemarchitecture evolves and a new service scenario emerges, the technicalsolutions provided in this application are also applicable to a similartechnical problem.

In the image recognition system 100 shown in FIG. 1, the display device10 and the image recognition device 40 may be integrated in a samedevice, or may be separate devices. When the display device 10 and theimage recognition device 40 are integrated in a same device, forexample, the display device 10 and the image recognition device 40 maybe respectively a display and a host of a computer, or the displaydevice 10 and the image recognition device 40 may be respectively adisplay screen and a mainboard of a notebook computer. The displaydevice 10 and the feedback signal collection device 20 may be integratedin a same device, or may be separate devices. The display device 10 andthe feedback signal collection device 20 may be integrated in a samedevice, which may be, for example, a virtual reality (VR) device. The VRdevice includes a module configured to collect the feedback signal ofthe observation object 30 and a display module configured to display theimage sequence.

FIG. 2 is a schematic structural diagram of a feedback signal collectiondevice 20 according to an embodiment of this application. As shown inFIG. 2, the feedback signal collection device 20 includes one or moredevice processors 201, a memory 202, a communications interface 203, areceiver 205, a transmitter 206, and an input/output module (including afeedback signal collecting module 207, an audio input/output module 208,a key input module 209, a display 210, and the like). It should be notedthat the display 210 may be a constituent part of the feedback signalcollection device 20, or may not be a constituent part of the feedbacksignal collection device 20. For better description, an example in whichthe display 210 is a constituent part of the feedback signal collectiondevice 20 is used in this embodiment of this application. The feedbacksignal collecting module 207 may be a sensor, and is configured tocollect a feedback signal of an observation object. These components maybe connected by using a bus 204 or in another manner. In FIG. 2, anexample in which the components are connected by using the bus is used.

The communications interface 203 may be used by the feedback signalcollection device 20 to communicate with another communications device,such as an image recognition device. Specifically, the image recognitiondevice may be the image recognition device 40 shown in FIG. 1.Specifically, the communications interface 203 may be a wiredcommunications interface 203, for example, a local access network (LAN)interface. The communications interface 203 is not limited to a wiredcommunications interface. The feedback signal collection device 20 mayfurther be configured with a long term evolution (LTE) (4G)communications interface, or may be configured with a 5G interface or acommunications interface of a future new air interface.

The transmitter 206 may be configured to perform transmissionprocessing, for example, signal modulation, on a signal output by thedevice processor 201. The receiver 205 may be configured to performreceiving processing, for example, signal demodulation, on a receivedsignal. In some embodiments of this application, the transmitter 206 andthe receiver 205 may be considered as a wireless modem. In the feedbacksignal collection device 20, there may be one or more transmitters 206and receivers 205.

In addition to the transmitter 206 and the receiver 205 shown in FIG. 2,the feedback signal collection device 20 may further include othercommunications components, for example, a GPS module, a Bluetoothmodule, and a Wi-Fi module. In addition to the foregoing describedwireless communication signal, the feedback signal collection device 20may further support other wireless communication signals, for example, asatellite signal and a short wave signal. In addition to wirelesscommunication, the feedback signal collection device 20 may further beconfigured with a wired network interface (for example, a LAN interface)to support wired communication.

The input/output module may be configured to implement interactionbetween the feedback signal collection device 20 and a user/externalenvironment, and may mainly include the feedback signal collectingmodule 207, the audio input/output module 208, the key input module 209,and the display 210. The feedback signal collecting module 207 isconfigured to collect a feedback signal of the observation object 30,and the display 210 may be used as the display device 10 in the imagerecognition system described in FIG. 1. Specifically, the input/outputmodule may further include a touchscreen, a sensor, and the like. Allthe input/output modules communicate with the device processor 201through a user interface 211.

The memory 202 is coupled to the device processor 201, and is configuredto store various software programs and/or a plurality of sets ofinstructions. Specifically, the memory 202 may include a high-speedrandom access memory, and may include a non-volatile memory, forexample, one or more disk storage devices, flash memory devices, orother non-volatile solid-state storage devices. The memory 202 may storean operating system (briefly referred to as a system in the following),for example, an embedded operating system such as ANDROID, IOS, WINDOWS,or LINUX. The memory 202 may further store a network communicationprogram. The network communication program may be used to communicatewith one or more additional devices and one or more image recognitiondevices 40. The memory 202 may further store a user interface program.The user interface program may vividly display content of an applicationprogram by using a graphical operation interface, and receive controloperations of a user for the application program by using input controlssuch as menus, dialog boxes, and keys.

In some embodiments of this application, the memory 202 may beconfigured to store an implementation program, on a side of the feedbacksignal collection device 20, of an image recognition method based onimage sequence presentation provided in one or more embodiments of thisapplication. Alternatively, the memory 202 may be configured to store animplementation program, on a side of the feedback signal collectiondevice 20, of an image presentation time determining method provided inone or more embodiments of this application. Alternatively, the memory202 may be configured to store an implementation program, on a side ofthe feedback signal collection device 20, of an image presentation timeadjustment method provided in one or more embodiments of thisapplication. Alternatively, the memory 202 may be configured to store animplementation program, on a side of the feedback signal collectiondevice 20, of an image recognition method provided in one or moreembodiments of this application. For implementation of the imagerecognition method provided in one or more embodiments of thisapplication, refer to subsequent embodiments.

The device processor 201 may be configured to read and execute acomputer readable instruction. Specifically, the device processor 201may be configured to invoke the program stored in the memory 202, andexecute an instruction included in the program. The program may be, forexample, the implementation program, on the side of the feedback signalcollection device 20, of the image recognition method based on imagesequence presentation provided in one or more embodiments of thisapplication; or the implementation program, on the side of the feedbacksignal collection device 20, of the image presentation time determiningmethod provided in one or more embodiments of this application; or theimplementation program, on the side of the feedback signal collectiondevice 20, of the image presentation time adjustment method provided inone or more embodiments of this application; or the implementationprogram, on the side of the feedback signal collection device 20, of theimage recognition method provided in one or more embodiments of thisapplication.

It may be understood that the feedback signal collection device 20 maybe the feedback signal collection device 20 in the image recognitionsystem 100 shown in FIG. 1, and may be implemented as a non-mobiledevice, a mobile device, a wearable device, a VR device, or the like.

It should be noted that the feedback signal collection device 20 shownin FIG. 2 is merely an implementation of this embodiment of thisapplication. In an actual application, the feedback signal collectiondevice 20 may further include more or fewer components. This is notlimited herein.

FIG. 3 is a schematic structural diagram of an image recognition device40 according to an embodiment of this application. As shown in FIG. 3,the image recognition device 40 includes one or more device processors401, a memory 402, a communications interface 403, a receiver 405, atransmitter 406, and an input/output module (including an audioinput/output module 407, a key input module 408, a display 409, and thelike). It should be noted that the display 409 may be a constituent partof the image recognition device 40, or may not be a constituent part ofthe image recognition device 40. For better description, an example inwhich the display 409 is a constituent part of the image recognitiondevice 40 is used in this embodiment of this application. Thesecomponents may be connected by using a bus 404 or in another manner. InFIG. 3, an example in which the components are connected by using thebus is used.

The communications interface 403 may be used by the image recognitiondevice 40 to communicate with another communications device, such as adisplay device. Specifically, the display device may be the displaydevice 10 shown in FIG. 1. Specifically, the communications interface403 may be a wired communications interface 403, for example, a localaccess network (LAN) interface. The communications interface 403 is notlimited to a wired communications interface. The device 40 may furtherbe configured with a long term evolution (LTE) (4G) communicationsinterface, or may be configured with a 5th generation (5G) interface ora communications interface of a future new air interface.

The transmitter 406 may be configured to perform transmissionprocessing, for example, signal modulation, on a signal output by thedevice processor 401. The receiver 405 may be configured to performreceiving processing, for example, signal demodulation, on a receivedsignal. In some embodiments of this application, the transmitter 406 andthe receiver 405 may be considered as a wireless modem. In the imagerecognition device 40, there may be one or more transmitters 406 andreceivers 405.

In addition to the transmitter 406 and the receiver 405 shown in FIG. 3,the image recognition device 40 may further include other communicationscomponents, for example, a GPS module, a Bluetooth module, and awireless fidelity (Wi-Fi) module. In addition to the foregoing describedwireless communication signal, the image recognition device 40 mayfurther support other wireless communication signals, for example, asatellite signal and a short wave signal. In addition to wirelesscommunication, the image recognition device 40 may further be configuredwith a wired network interface (for example, a LAN interface) to supportwired communication.

The input/output module may be configured to implement interactionbetween the image recognition device 40 and a user/external environment,and may mainly include the video input/output module 407, the key inputmodule 408, the display 409, and the like. The display 409 may be usedas the display device 10 in the image recognition system described inFIG. 1. Specifically, the input/output module may further include acamera, a touchscreen, a sensor, and the like. All the input/outputmodules communicate with the device processor 401 through a userinterface 410.

The memory 402 is coupled to the device processor 401, and is configuredto store various software programs and/or a plurality of sets ofinstructions. Specifically, the memory 402 may include a high-speedrandom access memory, and may include a non-volatile memory, forexample, one or more disk storage devices, flash memory devices, orother non-volatile solid-state storage devices. The memory 402 may storean operating system (briefly referred to as a system in the following),for example, an embedded operating system such as ANDROID, IOS, WINDOWS,or LINUX. The memory 402 may further store a network communicationprogram. The network communication program may be used to communicatewith one or more additional devices, one or more display devices, andone or more electroencephalogram collection devices 20. The memory 402may further store a user interface program. The user interface programmay vividly display content of an application program by using agraphical operation interface, and receive control operations of a userfor the application program by using input controls such as menus,dialog boxes, and keys.

In some embodiments of this application, the memory 402 may beconfigured to store an implementation program, on a side of the imagerecognition device 40, of an image recognition method based on imagesequence presentation provided in one or more embodiments of thisapplication. Alternatively, the memory 402 may be configured to store animplementation program, on a side of the image recognition device 40, ofan image presentation time determining method provided in one or moreembodiments of this application. Alternatively, the memory 402 may beconfigured to store an implementation program, on a side of the imagerecognition device 40, of an image presentation time adjustment methodprovided in one or more embodiments of this application. Alternatively,the memory 202 may be configured to store an implementation program, ona side of the image recognition device 40, of an image recognitionmethod provided in one or more embodiments of this application. Forimplementation of the image recognition method provided in one or moreembodiments of this application, refer to subsequent embodiments.

The device processor 401 may be configured to read and execute acomputer readable instruction. Specifically, the device processor 401may be configured to invoke the program stored in the memory 402, andexecute an instruction included in the program. The program may be, forexample, the implementation program, on the side of the imagerecognition device 40, of the image recognition method based on imagesequence presentation provided in one or more embodiments of thisapplication; or the implementation program, on the side of the imagerecognition device 40, of the image presentation time determining methodprovided in one or more embodiments of this application; or theimplementation program, on the side of the image recognition device 40,of the image presentation time adjustment method provided in one or moreembodiments of this application; or the implementation program, on theside of the image recognition device 40, of the image recognition methodprovided in one or more embodiments of this application.

It may be understood that the image recognition device 40 may be theimage recognition device 40 in the image recognition system 100 shown inFIG. 1, and may be implemented as a non-mobile device or a mobiledevice.

It should be noted that the image recognition device 40 shown in FIG. 3is merely an implementation of this embodiment of this application. Inan actual application, the image recognition device 40 may furtherinclude more or fewer components, which is not limited herein.

Currently, times for presenting all images in an image sequence based ona rapid serial visual representation paradigm on the display device 10is generally uniform, and the presentation times may be determinedaccording to experience or an experiment. However, because the brain ofan observation object is prone to fatigue and attention resources of thebrain of the observation object are limited, a miss detection rate ofbrain-computer combination image recognition is still high, resulting inlow efficiency of brain-computer collaboration image recognition.

Based on the schematic diagram of the architecture of the imagerecognition system in FIG. 1, an embodiment of this application providesa brain-computer combination image recognition method based on imagesequence presentation. In the brain-computer combination imagerecognition method based on image sequence presentation, the imagerecognition device determines or adjusts an image presentation timebased on at least one of a first recognition probability obtainedthrough calculation by using a computer vision algorithm and a fatiguestate parameter corresponding to the image. A smaller first recognitionprobability indicates a longer presentation time and a longer watchingtime for the observation object. A larger fatigue state parameter of theobservation object indicates a longer presentation time and a longerwatching time for the observation object. The brain-computer combinationimage recognition method based on image sequence presentation makesbetter use of attention resources of the brain of the observation objectin a time dimension to recognize an image, and allocates more attentionresources of the observation object in the time dimension to an imagewith relatively great uncertainty. Therefore, a miss detection rate ofimage recognition can be reduced, and efficiency of brain-computercombination image recognition is improved.

Main inventive principles in this application may include: The imagerecognition device calculates a first recognition probability of atarget image in an image sequence by using a computer vision algorithm,and obtains a fatigue state parameter corresponding to the target image.The image recognition device determines or adjusts a presentation timeof the target image on the display device based on at least one of thefirst recognition probability corresponding to the target image and thefatigue state parameter corresponding to the target image. A policy ofdetermining or adjusting, by the image recognition device, thepresentation time may be: When the first recognition probability issmaller, and the fatigue state parameter of the observation object islarger, a longer presentation time is set or adjusted. In other words,when uncertainty of recognition by using the computer vision algorithmis greater, or the fatigue state parameter of the observation object islarger, a time for presenting the image for the brain to recognize islonger. In this way, a miss detection rate can be reduced. On thecontrary, when the first recognition probability is larger, and thefatigue state parameter of the observation object is smaller, and ashorter presentation time is set or adjusted. When uncertainty ofrecognition by using the computer vision algorithm is relatively small,or the observation object is not fatigued, a relatively long brainrecognition time is not required. This can reduce the fatigue caused bybrain recognition and reduce the miss detection rate.

For example, when a first recognition probability c of an image A meets0.7<c≤0.9, a probability that the image recognition device recognizes,by using the computer vision algorithm, that the image A in the imagesequence includes a preset image feature is relatively high. In otherwords, the computer vision algorithm recognizes the image A with smalluncertainty. In this case, it is unnecessary to set a long time for thebrain of the observation object to recognize the image A, and the imagerecognition device may correspondingly set a presentation time of theimage A to 0.1 s. When a first recognition probability c of an image Bin the image sequence meets 0.5<c≤0.7, that is, a probability that theimage recognition device recognizes, by using the computer visionalgorithm, that the image B includes the preset image feature is smallerthan the previous image, the image recognition device may set a longerpresentation time for the image B, for example, 0.2 s. When a firstrecognition probability c of an image C in the image sequence meets0.3<c≤0.5, a probability of recognizing, by using the computer visionalgorithm, that the image C includes the preset image feature isrelatively small. In other words, the computer vision algorithmrecognizes the image with great uncertainty, and the image recognitiondevice may set a relatively long time for the brain of the observationobject for recognizing, so as to reduce a miss detection rate. Forexample, a presentation time of the image C may be set to 0.4 s. In theforegoing method, more attention resources of the observation object inthe time dimension may be allocated to an image with relatively greatuncertainty. In this way, a miss detection rate of image recognition canbe reduced, and efficiency of brain-computer collaboration imagerecognition is improved.

For another example, when a fatigue state parameter f of the observationobject when the observation object observes the image A is fm, and isrelatively small, it indicates that the observation object hasrelatively active brain thinking. In this case, the observation objectcan recognize, within a relatively short observation time, whether theimage includes the preset image feature. Therefore, the imagerecognition device may set the presentation time of the image A to 0.1s. When a fatigue state parameter f of the observation object when theobservation object observes the image B is fm′, and is relatively large,it indicates that the observation object has relatively slow brainthinking. In this case, the observation object requires a relativelylong observation time to recognize whether the image includes the presetimage feature. Therefore, the image recognition device may set thepresentation time of the image B to 0.3 s. Determining an imagepresentation time based on the fatigue state parameter of theobservation object can reduce miss detection caused by brain fatigue ofthe observation object, thereby reducing the miss detection rate.

Based on the foregoing main invention principles, the followingdescribes several embodiments provided in this application.

FIG. 4 is a schematic flowchart of a brain-computer combination imagerecognition method based on image sequence presentation according to anembodiment of this application. In this method, an image recognitiondevice sets a presentation time of an image based on a duration impactparameter. As shown in FIG. 4, the brain-computer combination imagerecognition method based on image sequence presentation includes but isnot limited to the following operations S101 to S104.

S101. The image recognition device sets a presentation time sequencecorresponding to an image sequence, where the presentation time sequenceincludes at least two unequal presentation times.

The foregoing image sequence may be an image sequence based on RSVP, andmay include N images, where N is a positive integer.

S102. The image recognition device processes the image sequence by usinga computer vision algorithm, to obtain a computer vision signalcorresponding to each image in the image sequence.

S103. The image recognition device obtains a feedback signal that isgenerated when an observation object watches the image sequencedisplayed in the presentation time sequence and that corresponds to eachimage in the image sequence.

S104. The image recognition device fuses, for each image in the imagesequence, a corresponding computer vision signal and a correspondingfeedback signal to obtain a target recognition signal of each image inthe image sequence.

Operation S102 may be performed before step S101, or may be performedafter operation S103.

A presentation time of an image is used to indicate a time period from apresentation start moment of the image to a presentation start moment ofa next adjacent image. The presentation time of the image may be a timeperiod during which the image is displayed on a display device, or mayinclude the time period during which the image is displayed on thedisplay device and a time period from a moment when the display devicestops displaying the image to a moment when the display device starts todisplay a next adjacent image. The time period from a moment when thedisplay device stops displaying the image to a moment when the displaydevice starts to display a next adjacent image may be used for theobservation object to rest.

Specifically, FIG. 5 is a schematic diagram of an image presentationtime according to an embodiment of this application. As shown in FIG. 5,a presentation time of an image may include a sum of a time fordisplaying the image on the display device and a black screen time afterthe image is displayed. The black screen time may be used by theobservation object to take a rest after observing the image. As shown inFIG. 5, the image sequence may include an image 1, an image 2, an image3, an image 4, and an image 5. In the image sequence, a time fordisplaying the image 1 on the display screen is 200 ms, and a blackscreen time of the display device after the image 1 is displayed is 100ms. Therefore, a presentation time of the image 1 may be 300 ms. A blackscreen time corresponding to an image may also be set to 0.

It may be understood that, a presentation time of an image in thisembodiment of this application is not limited to the foregoingdefinition. For example, a presentation time may also be a time when theimage is displayed on the display device. Specifically, in theembodiment described in FIG. 5, the presentation time of the image 1 mayalso be defined as 200 ms.

In this embodiment of this application, presentation times of all imagesin the image sequence are no longer completely the same, and thespecified presentation time sequence corresponding to the image sequencemay include at least two unequal presentation times. A differencebetween any two presentation times of the at least two unequalpresentation times is k×Δ, where k is a positive integer, and Δ is apreset time period value. The at least two unequal presentation timesare set to improve accuracy of recognizing each image in the imagesequence by the observation object. The time period value Δ may be avalue between 10 ms to 100 ms. In an embodiment, the time period value Δmay be a value between 50 ms to 100 ms.

For example, the image sequence includes 100 images, Δ may be 50 ms, andpresentation times of the images in the image sequence are in ascendingorder according to an arithmetic sequence. To be specific, apresentation time of the i^(th) image is [100+(i−1)*50] ms, and i is aninteger that meets 1≤i≤100. In the image sequence, different imagescorrespond to different presentation times. This example is merely usedto explain this embodiment of this application without constituting anylimitation.

In an embodiment, the at least two unequal presentation times in thepresentation time sequence and the presentation time difference k×Δ maybe determined based on a duration impact parameter of each image in theimage sequence. The setting a presentation time sequence correspondingto an image sequence may include: determining a presentation timecorresponding to each image in the image sequence based on the durationimpact parameter, so as to obtain the presentation time sequencecorresponding to the image sequence. The duration impact parameterincludes at least one of a first recognition probability and a fatiguestate parameter.

The first recognition probability is a probability, obtained by theimage recognition device through calculation by using a computer visionalgorithm, that an image includes a preset image feature. A feedbacksignal generated when the observation object observes that an imageincludes the preset image feature conforms to a specific feedback signalfeature. The specific feedback signal feature may be, for example, afeature met by a P300 electroencephalogram signal.

The computer vision signal obtained through calculation by using thecomputer vision algorithm is the first recognition probability or animage feature of an image. The target recognition signal may be aprobability that an image includes the preset image feature, and is usedto determine whether the image includes the preset image feature, so asto perform target image recognition.

In this embodiment of this application, a presentation time T(c) isinversely correlated with a first recognition probability c. A largerfirst recognition probability of any image i in the image sequenceindicates a shorter presentation time that the image recognition devicesets for the image i. A larger fatigue state parameter corresponding tothe image i indicates a longer presentation time that the imagerecognition device sets for the image i. When the probability,determined by using the computer vision algorithm, that the image iincludes the preset image feature is relatively small, uncertainty ofrecognition performed by the image recognition system on the image isrelatively great, and a relatively long electroencephalogram recognitiontime for the image i may be set. In this way, more attention resourcesof the observation object in the time dimension are allocated to theimage with relatively great uncertainty, and the miss detection rate ofthe image recognition system can be reduced. On the contrary, a largerfatigue state parameter corresponding to the image i indicates a longerpresentation time of the image i, that is, when the fatigue stateparameter of the observation object is larger, a time for presenting theimage i for the brain to recognize is longer, thereby reducing a missdetection rate. The presentation time of each image in the imagesequence is set according to a principle that the presentation time isinversely correlated with the first recognition probability and ispositively correlated with the fatigue state parameter, attentionresources of the observation object can be allocated to a “weak part” ofthe image recognition system in the time dimension, and a program timeis properly set based on a fatigue degree of the observation object, sothat recognition efficiency of the image recognition system can beimproved and the miss detection rate can be reduced compared withsetting a uniform presentation time.

The brain-computer combination image recognition based on image sequencepresentation is specifically described in three parts: 1. This partdescribes how to determine, based on the duration impact parameter, thepresentation time sequence corresponding to the image sequence when theduration impact parameter includes different parameters. 2. This partdescribes setting a presentation time of each image in the imagesequence in real time or in advance. 3. This part describesmulti-observation object recognition, multi-round recognition, and abrain-computer fusion weight. Descriptions are separately provided inthe following:

1. How to determine, based on the duration impact parameter, thepresentation time sequence corresponding to the image sequence?

The duration impact parameter includes at least one of a firstrecognition probability and a fatigue state parameter, and there arethree cases: (1) The duration impact parameter includes the firstrecognition probability; (2) The duration influence parameter includesthe fatigue state parameter; and (3) The duration impact parameterincludes the first recognition probability and the fatigue stateparameter. The following specifically describes how to determine, basedon the duration impact parameter, the presentation time sequencecorresponding to the image sequence in the foregoing three cases.

(1) The duration impact parameter includes the first recognitionprobability c.

A correspondence between the first recognition probability c and thepresentation time T may be determined by using a fitting formula, or maybe determined by using a mapping table. Descriptions are separatelyprovided in the following:

a. Determining the Correspondence Between the First RecognitionProbability c and the Presentation Time T in a Formula Fitting Manner

In an embodiment, the duration impact parameter may include the firstrecognition probability c. T(c) may be obtained after n-order linearfitting, nonlinear fitting, or the like is performed on c. For example,n-order linear fitting is used to obtain:

$\begin{matrix}{{T(c)} = {\sum\limits_{t = 0}^{n}{a_{j}c^{t}}}} & \left( {1\text{-}2} \right)\end{matrix}$

where T(c) is the presentation time, c is the first recognitionprobability, c is a real number satisfying 0≤c≤1, n is an order at whichT(c) fits c, n is an integer greater than 0, t is an integer satisfying−n≤t≤n, and a_(t) is a coefficient of c^(t).

In an embodiment, T(c) may be obtained through fitting based on aprobability threshold and a presentation time threshold, and thefollowing gives a specific description:

First, the probability threshold is explained as follows: A minimumprobability threshold c1 and a maximum probability threshold c2 mayexist for the first recognition probability c. When the firstrecognition probability c obtained through calculation by using thecomputer vision algorithm is less than or equal to c1, the brain of theobservation object does not need to perform electroencephalogramrecognition, and it may be directly determined, by using the computervision algorithm, that an image in the image sequence does not includethe preset image feature. When the first recognition probability c isgreater than or equal to c2, brain recognition of the observation objectis not required either, and it may be directly determined, by using thecomputer vision algorithm, that an image in the image sequence includesthe preset image feature. If the image i meets either of the foregoingtwo cases, the image i does not need to be placed in the image sequencefor the observation object to watch to perform brain image recognition,that is, whether the image i includes the preset image feature may bedirectly recognized by using the computer vision algorithm. The image imay be any image in the image sequence.

Second, the presentation time threshold is explained as follows: Aminimum presentation time threshold T1 and a maximum presentation timethreshold T2 may exist for a presentation time T of an image in theimage sequence. T1 and T2 may be set based on a physiological feature ofimage recognition performed when the observation object watches theimage sequence. To be specific, it needs to be ensured that theobservation object can recognize, within the presentation time T1,whether the image includes the preset image feature; in addition, thepresentation time cannot be too long, to avoid wasting brain attentionresources in the time dimension, and therefore the maximum presentationtime threshold T2 may be set.

In an embodiment, T(c) may be obtained after fitting is performed on cby using c1, c2, T1, and T2. In an embodiment, T(c) may be obtainedafter fitting is performed on c by using (c1, T2) and (c2, T1). When afirst recognition probability of an image is the minimum probabilitythreshold c1, the image recognition device sets a presentation time ofthe image to the maximum presentation time threshold T2. When a firstrecognition probability of an image is the maximum probability thresholdc2, the image recognition device sets a presentation time of the imageto the minimum presentation time threshold T1. The fitting may ben-order linear fitting, or may be nonlinear fitting, for example, leastsquare fitting.

Specifically, for example, n-order linear fitting may be used to obtain:

$\begin{matrix}{{T(c)} = {T_{1} + {\left( {c - {c\; 1}} \right)^{n} \cdot \frac{{T\; 1} - {T\; 2}}{\left( {{c\; 2} - {c\; 1}} \right)^{n}}}}} & \left( {1\text{-}3} \right)\end{matrix}$

In an embodiment, when a first recognition probability of an image isthe minimum probability threshold c1, a presentation time of the imagemay not be set to the maximum presentation time threshold T2, but is setto, for example, be slightly less than T2. Likewise, when a firstrecognition probability of an image is the maximum probability thresholdc2, a presentation time of the image may not be set to the minimumpresentation time threshold T1, but is set to, for example, be slightlygreater than T2. In an embodiment, T(c) may also be obtained afterlinear fitting or nonlinear fitting is performed on c by using anotherparameter. This is not limited in this application. A fittingrelationship between T(c) and c may also be obtained by adding acoefficient, or adding a constant, or the like on the basis of theformula (1-3).

In an embodiment, the probability threshold c1 may also be a minimumvalue of probabilities that the preset image feature is included in around of image sequence prediction performed by using the computervision algorithm, and the probability threshold c2 may also be a maximumvalue of probabilities that the preset image feature is included in around of image sequence prediction performed by using the computervision algorithm.

In the foregoing formula (1-2) and formula (1-3), in a value range ofthe first recognition probability c, T(c) is a monotonically decreasingfunction, that is, a larger first recognition probability of an imageindicates a shorter presentation time of the image on the displaydevice.

In an embodiment, after the presentation time of each image in the imagesequence is determined based on the fitting relationship between c andT, the presentation time of the corresponding image may be furtheradjusted based on a fatigue state parameter of the observation object.

In other words, the fatigue state parameter is used as an independentvariable for adjusting the presentation time of the image i. Inaddition, the fatigue state parameter is positively correlated with theimage presentation time. In other words, when the fatigue stateparameter is larger, the presentation time of the image i may beadjusted to be longer. An adjusted presentation time T′(c, f) may bedetermined based on a fitting relationship between the presentation timeof the image and f. A relationship between T′(c, f), and f and c may beobtained through n-order linear fitting, or may be obtained throughnon-linear fitting. This is not limited in this embodiment of thisapplication. A value of n may be a positive integer greater than 0. Theimage i may be any image in the image sequence.

Because the observation object is not fatigued until the image sequenceis displayed for a period of time. In other words, impact of the fatiguestate parameter f on the presentation time T of the image i usually lagsbehind impact of the first recognition probability c on T. When arelationship between the presentation time T of the image and the firstrecognition probability c is determined, the first recognitionprobability c may be first used as an independent variable to performfitting on T, and then impact of the fatigue state parameter f isimposed on the fitting result. A method for imposing the impact of thefatigue state parameter f may be adding an increment T1(f) to T(c)determined in the formula (1-2) or (1-3), so as to obtain a presentationtime T′(c, f) considering the impact of the fatigue state parameter. Forthe fitting manner, the first recognition probability c is used as anindependent variable to perform fitting on T to obtain the formula (1-2)or the formula (1-3), and

$\begin{matrix}{{T^{\prime}\left( {c,f} \right)} = {{{T(c)} + {T\; 1(f)}} = {{T\; 1(f)} + {\sum\limits_{t = 0}^{n}{a_{t}c^{t}}}}}} & \left( {1\text{-}4} \right)\end{matrix}$

where T′(c, f) is a presentation time of an image that considers c andf, and T1(f) is the impact of the fatigue state parameter f on thepresentation time of the image. T1(f) may be a positive value, 0, or anegative value. T1(f) may also be a constant. Alternatively, T1(f) maybe T1(f)=T(c)*y % where y is a positive value, 0, or a negative value,and y may be a constant.

Certainly, if the fatigue state parameter is not considered, T(c) may bedetermined by using a mapping table, and an increment T1(f) is added toimplement impact of the fatigue state parameter on the presentationtime. For determining a correspondence between the presentation timeT(c) and the first recognition probability c by using a mapping table,refer to the following specific description. Details are not describedherein again. In addition to being obtained by using the fittingformula, T1(f) may also be determined by using a mapping table.

For the increment T1(f) generated for the presentation time due to thefatigue state parameter f, the following is described:

{circle around (1)} The increment T1(f) of the presentation time may beobtained through fitting based on a fatigue threshold:

First, the fatigue threshold is explained as follows: When it isdetected that the fatigue state parameter f of the observation object isgreater than or equal to a first fatigue threshold f2, the imagerecognition device may determine that the observation object isexcessively fatigued and needs to rest. The first fatigue threshold f2may be considered as a maximum fatigue state parameter of theobservation object that can be tolerated by the image recognitionsystem. When it is detected that the fatigue state parameter of theobservation object is greater than or equal to the first fatiguethreshold f2, the image recognition device may suspend displaying theimage sequence to be observed by the observation object. When it isdetected that the fatigue state parameter f of the observation object isless than or equal to a second fatigue threshold f1, the imagerecognition device may determine that the observation object canre-perform electroencephalogram image recognition, and may re-enable theimage sequence for image recognition by using an electroencephalogramsignal. f2 may be greater than or equal to f1. When the fatigue stateparameter is between the first fatigue threshold f2 and the secondfatigue threshold f1, a larger fatigue state parameter indicates alonger image presentation time. This reduces a case in which detectionof an image is missed due to fatigue of the observation object, and mayreduce an increase in a miss detection rate caused by the fatigue stateparameter.

Specifically, for example, T1(f) may be related to T(c), and T1(f) maybe obtained through one-order linear fitting:

$\begin{matrix}{{T\; 1(f)} = {{T(c)}*\frac{f - {f\; 1}}{{f\; 2} - {f\; 1}}}} & \left( {1\text{-}5} \right)\end{matrix}$

In an embodiment, T1(f) may be obtained through non-linear fitting, anda coefficient in a fitting formula is determined based on the firstfatigue threshold f2 and the second fatigue threshold f1. For example,fitting is performed by using a least square method. This is not limitedin this embodiment of this application.

T′(c, f) may be obtained by using the formula (1-4) and the formula(1-5):

$\begin{matrix}{{T^{\prime}\left( {c,f} \right)} = {{{T(c)} + {T\; 1(f)}} = {{T(c)}*\left( {1 + \frac{f - {f\; 1}}{{f\; 2} - {f\; 1}}} \right)}}} & \left( {1\text{-}6} \right)\end{matrix}$

{circle around (2)} The increment T1(f) of the presentation time mayalso be related to the first recognition probability:

When the first recognition probability is relatively large, it indicatesthat a probability that the computer vision algorithm recognizes animage as a specific object is relatively large. Therefore, there is noneed to add a long image presentation time for electroencephalogramrecognition. When the first recognition probability is relatively small,it indicates that a probability that the computer vision algorithmrecognizes an image as a specific object is relatively small. Therefore,a relatively long image presentation time needs to be added for theobservation object to recognize the image as the specific object. Basedon the foregoing principle, the increment T1(f) of the presentation timemay be set to be inversely correlated with the first recognitionprobability c.

In an embodiment, a coefficient affected by the first recognitionprobability c or an increment affected by the first recognitionprobability c may be added on the basis of the formula (1-6), toincrease coefficient examples:

$\begin{matrix}{{T1(f)} = {{T(c)}*\frac{f - {f1}}{{f2} - {f1}}*{x(c)}}} & \left( {1\text{-}7} \right)\end{matrix}$

where x(c) is an impact coefficient of the first recognition probabilityc on the increment T1(f) of the presentation time. A larger firstrecognition probability c indicates a smaller x(c) and a smaller firstrecognition probability c indicates a larger x(c).

A relationship between c and x(c) may be a linear fitting relationship,or may be another fitting relationship. This is not limited in thisapplication. x(c) may be determined based on c by querying a table.Referring to Table 1, when T1(f) is a positive value in the formula(1-5), Table 1 is an example of a correspondence between the firstrecognition probability c and the impact coefficient x(c).

TABLE 1 Example of a correspondence between a first recognitionprobability c and an impact coefficient x(c) (T1(f) is a positive value)c (0.1, 0.3] (0.3, 0.5] (0.5, 0.7] (0.7, 0.9] x(c) 1.5 1.2 0.9 0.6

As shown in Table 1, if T1(f) is a positive value, when firstrecognition probabilities c are within intervals (0.1, 0.3], (0.3, 0.5],(0.5, 0.7], and (0.7, 0.9], values of x(c) are respectively 1.5, 1.2,0.9, and 0.6. A larger first recognition probability c indicates asmaller x(c), and a smaller first recognition probability c indicates alarger x(c).

It may be understood that this example is merely used to explain thisembodiment of this application without constituting any limitation.

{circle around (3)} A working process of the image recognition systemwhen the observation object is fatigued

When it is detected that the fatigue state parameter of the observationobject is greater than or equal to the first fatigue threshold, theimage recognition device may determine that the fatigue state parameterof the observation object significantly affects electroencephalogramimage recognition. Therefore, the image recognition device may controlto suspend a display process of the image sequence. The first fatiguethreshold may be used as a threshold for the image recognition device todetermine whether the fatigue state parameter of the observation objectis suitable for continuing electroencephalogram image recognition.

When it is detected that the fatigue state parameter of the observationobject is greater than or equal to the first fatigue threshold f2, theimage recognition device may further control to suspend displaying ofthe image sequence, and obtain images whose first recognitionprobabilities are greater than or equal to a first probability thresholdin the image sequence. When it is detected that the fatigue stateparameter of the observation object is less than or equal to the secondfatigue threshold, the images whose first recognition probabilities aregreater than or equal to the first probability threshold in the imagesequence are displayed in sequence in terms of time based on apresentation time of one image in the image sequence. The firstprobability threshold may be flexibly adjusted based on a performancerequirement of the image recognition system in an actual application,for example, the first probability threshold is determined based on asensitivity requirement of the system.

In other words, when the fatigue state parameter of the observationobject reaches a specific threshold, the image recognition device maysuspend the electroencephalogram image recognition by the observationobject, so that the observation object can take a rest. In a process inwhich the observation object is resting, the image recognition devicemay select, by using the first recognition probability of each image inthe image sequence that is obtained through calculation by using thecomputer vision algorithm, images whose first recognition probabilitiesare greater than the first probability threshold. The selected imagesare used as a new image sequence for brain-computer combination imagerecognition. When the fatigue state parameter of the observation objectis less than or equal to the second fatigue threshold, it indicates thatthe observation object has finished rest, and may continue toparticipate in electroencephalogram image recognition. In this case, theimage recognition device may control display of the selected imagesequence, so that the image recognition device performs brain-computercombination image recognition. The first fatigue threshold is greaterthan or equal to the second fatigue threshold. Values of the firstfatigue threshold and the second fatigue threshold may be determinedbased on a physiological feature of the observation object. When theobservation object is fatigued, displaying of the image sequence issuspended through control, so that the observation object rests, and animage with a relatively high first recognition probability is selectedduring this period. When the observation object has finished rest,brain-computer combination image recognition is performed on theseimages. The foregoing process can improve recognition efficiency of theimage recognition system.

In addition, when the image recognition system performs brain-computercombination image recognition, if it is detected that the fatigue stateparameter of the observation object is between the first fatiguethreshold and the second fatigue threshold, that is, the fatigue stateparameter f of the observation object falls within [f2, f1], the imagerecognition device may further increase, based on the fatigue stateparameter f of the observation object, a presentation time of an imagedisplayed after the current moment in the image sequence. If the imagerecognition device has determined, according to operation S401 in themethod described in FIG. 4, that the presentation time of the image A isTn, when it is detected that the fatigue state parameter of theobservation object is between the first fatigue threshold and the secondfatigue threshold, the presentation time of the image A displayed afterthe current moment in the image sequence may be set to Tn+Tk.Alternatively, the presentation time of the image A may be set toTn×(1+p %), or the presentation time of the image A may be set to Tn×b.Tk indicates a duration increment, p indicates a duration incrementpercentage, and b indicates a duration increment multiple. Tk, p, and bmay be constants or variables.

Actually, when the fatigue state parameter of the observation object isbetween the first fatigue threshold and the second fatigue threshold,the observation object has already started to be fatigued, but stilldoes not reach a fatigue threshold for controlling to stop imagesequence display for electroencephalogram recognition. In this case, apresentation time of an image displayed after the current moment in theimage sequence is further increased. A longer presentation time resultsin smaller impact on accuracy of recognition by using anelectroencephalogram signal of the observation object. This can reduce acase of miss detection caused by fatigue of the observation object, sothat a miss detection rate can be reduced.

{circle around (4)} The fatigue state parameter may be obtained throughreal-time measurement for the observation object by using a sensor, ormay be predicted by using a fatigue rule.

The fatigue rule may be embodied in a manner of a mapping table, or maybe embodied in a manner of a fitting formula. The fatigue rule may beused to indicate a change rule of the fatigue state parameter of theobservation object based on a quantity of images observed by theobservation object, or the fatigue rule is used to indicate a changerule of the fatigue state parameter of the observation object based on aduration spent by the observation object for image observation.Descriptions are separately provided in the following:

The fatigue rule may include a second mapping table. When the fatiguerule is used to indicate a change rule of the fatigue state parameter ofthe observation object based on a quantity of images observed by theobservation object, refer to Table 2. Table 2 is an example of thesecond mapping table provided in this embodiment of this application.

TABLE 2 Example of a second mapping table S 1 2 3 . . . f 1.0 1.03 1.1 .. .

As shown in Table 2, the second mapping table includes a plurality ofquantities S of observed images and fatigue state parameters fcorresponding to the plurality of quantities S of observed images. Whenthe quantities of images observed by the observation object are 1, 2, 3,and so on, corresponding fatigue state parameters may be 1, 1.03, 1.1,and so on. Predicting, according to the fatigue rule, a fatigue stateparameter corresponding to each image in the image sequence includes:finding, from the second mapping table according to a quantity of imagesdisplayed before each image in the image sequence, the fatigue stateparameter corresponding to each image in the image sequence. In theimage sequence, a quantity of images displayed before a specific imageis a quantity of observed images corresponding to the specific image.

The second mapping table may also include a plurality of ranges ofquantities of observed images and fatigue state parameters fcorresponding to the plurality of ranges of quantities of observedimages. To search for a fatigue state parameter corresponding to aspecific image in the image sequence, a range of quantities of observedimages to which a quantity of images that have been displayed before theimage belongs may be first searched for, and then a fatigue stateparameter corresponding to the range of quantities of observed images isfound from the second mapping table.

In an embodiment, the fatigue rule may include a second mapping table.When the fatigue rule is used to indicate a change rule of the fatiguestate parameter of the observation object based on a duration spent bythe observation object for image observation, refer to Table 3. Table 3is an example of another second mapping table provided in thisembodiment of this application.

TABLE 3 Example of another second mapping table t/s 10 10.5 11 . . . f1.0 1.05 1.2 . . .

As shown in Table 3, the second mapping table includes a plurality ofdurations t spent for image observation and fatigue state parameters fcorresponding to the plurality of durations t spent for imageobservation. When the durations t spent by the observation object forimage observation are 10 s, 10.5 s, 11 s, and so on, correspondingfatigue state parameters may be 1, 1.05, 1.2, and so on. The predicting,according to the fatigue rule, a fatigue state parameter correspondingto each image in the image sequence includes: predicting, according to aquantity S of images displayed before each image in the image sequence,a duration spent by the observation object for image observation wheneach image in the image sequence is being observed, where a durationspent by the observation object for image observation when an image isbeing observed is t=S×ts, ts is a predicted average presentation time ofeach image in the image sequence; and finding, from the second mappingtable, the fatigue state parameter corresponding to each image in theimage sequence according to the duration spent by the observation objectfor image observation when each image in the image sequence is beingobserved.

The second mapping table may also include a plurality of ranges ofdurations spent for image observation and fatigue state parameters fcorresponding to the plurality of ranges of durations spent for imageobservation. To search for a fatigue state parameter corresponding to aspecific image in the image sequence, a range of durations spent forimage observation to which a duration of images that have been displayedbefore the image belongs may be first searched for, and then a fatiguestate parameter corresponding to the range of durations spent for imageobservation is found from the second mapping table.

It may be understood that this example of the second mapping table ismerely used to explain this embodiment of this application withoutconstituting any limitation.

The fatigue rule may also be a fitting relationship between a fatiguestate parameter and a quantity of observed images, or a fittingrelationship between a fatigue state parameter and a time spent forimage observation. The fitting relationship may be linear, or may benon-linear. The fitting relationship may be an objective law related tothe observation object, for example, a fatigue rule of watching an imageby a person, a fatigue rule of watching an image by a gorilla, or thelike.

The fitting relationship may also be obtained by training one or moreobservation objects by using a plurality of samples. Each sample in theplurality of samples may be a combination of a quantity of observedimages and a fatigue state parameter, and a mapping relationship that isobtained through training and that reflects a fatigue rule is a fittingrelationship between a fatigue state parameter and a quantity ofobserved images. Each sample in the plurality of samples may also be acombination of a duration spent for image observation and a fatiguestate parameter, and a mapping relationship that is obtained throughtraining and that reflects a fatigue rule is a fitting relationshipbetween a fatigue state parameter and a duration spent for imageobservation. Specifically, for example, a fatigue rule for a group ofpeople may be trained, and the group of people may be, for example,criminal investigators. A model reflecting the fatigue rule for thiskind of people can be obtained through training by using a large amountof sample data, and the model is used to predict a change rule of afatigue state parameter of this kind of people.

b. Determining the Correspondence Between the First RecognitionProbability c and the Presentation Time by Using a Mapping Table

In an implementation, the image recognition device sets a presentationtime T(c) of an image in the image sequence according to a firstrecognition probability c of the image, or may determine thepresentation time of the image by using a mapping table. Specifically,the image recognition device may find, from the first mapping tableaccording to a first recognition probability c of an image in the imagesequence, a presentation time T corresponding to the first recognitionprobability c of the image. The first mapping table may be prestored inthe image recognition device. The first mapping table includes aplurality of probabilities and presentation times respectivelycorresponding to the plurality of probabilities. Then, the imagerecognition device sets a presentation time of an image to apresentation time corresponding to a first recognition probability.

The probability in the first mapping table may be a probability value,or may be a probability interval. When the probability in the mappingtable is a probability interval, a presentation time corresponding to afirst recognition probability is a presentation time corresponding to aprobability interval to which the first recognition probability belongs.For example, Table 4 is an example of a first mapping table provided inthis embodiment of this application.

TABLE 4 Example of a first mapping table c (0.1, 0.3] (0.3, 0.5] (0.5,0.7] (0.7, 0.9] T(c)/s 0.4 0.3 0.2 0.1

As shown in Table 4, when first recognition probabilities fall withinintervals (0.1, 0.3], (0.3, 0.5], (0.5, 0.7], and (0.7, 0.9]), values ofpresentation times of images are respectively 0.4 s, 0.3 s, 0.2 s, and0.1 s. When the first recognition probability is smaller, a longer imagepresentation time is set.

When the probability in the mapping table is a probability value, forexample, Table 5 is an example of another first mapping table providedin this embodiment of this application.

TABLE 5 Example of another first mapping table c 0.1 0.2 0.3 0.4 0.5 0.60.7 0.8 0.9 T(c)/s 0.4 0.36 0.33 0.3 0.25 0.2 0.16 0.13 0.1

As shown in Table 5, when first recognition probabilities arerespectively 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, and 0.9, values ofpresentation times of images are respectively 0.4 s, 0.36 s, 0.33 s, 0.3s, 0.25 s, 0.2 s, 0.16 s, 0.13 s, and 0.1 s. When a first recognitionprobability is smaller, a longer image presentation time is set, thatis, a time for presenting, on the display device, the image forelectroencephalogram recognition by the observation object is longer.

It may be understood that this example of the first mapping table ismerely used to explain this embodiment of this application withoutconstituting any limitation.

(2) The duration impact parameter includes the fatigue state parameterf.

In general, a presentation time of an image may be inversely correlatedwith a fatigue state parameter of the observation object. In otherwords, a larger fatigue state parameter indicates a longer presentationtime of the image.

A relationship between the presentation time T of the image and thefatigue state parameter f of the observation object may be determined byusing a mapping table. A specific mapping table may be similar to thefirst mapping tables shown in Table 4 and Table 5, and details are notdescribed herein again.

T(f) may be obtained after m-order linear fitting, nonlinear fitting, orthe like is performed on f. For example, m-order linear fitting is usedto obtain:

$\begin{matrix}{{T(f)} = {\sum\limits_{k = 0}^{m}{a_{k}f^{k}}}} & \left( {1\text{-}8} \right)\end{matrix}$

T(f) is a presentation time, f is a fatigue state parameter, m is anorder at which T(f) fits f, m is a positive integer greater than 0, k isan integer satisfying −m≤k≤m, and a_(k) is a coefficient of f^(k).

In an embodiment, T(c) may be obtained through fitting based on afatigue threshold and a presentation time threshold. A fittingrelationship between the presentation time T of the image and thefatigue state parameter f of the observation object may also be linearfitting, nonlinear fitting, or the like, which is not limited in thisapplication.

A minimum fatigue threshold f1 and a maximum fatigue threshold f2 areexplained as follows: When a fatigue state parameter corresponding to atarget image is greater than or equal to f2, it indicates that theobservation object is already very fatigued and needs to rest, and theimage recognition device controls to stop displaying the image sequenceto perform brain image recognition. When a fatigue state parametercorresponding to an image in the image sequence is less than or equal tof1, it indicates that the observation object has finished rest, andbrain image recognition may continue to be performed, and the imagesequence may start to display.

Specifically, the presentation time T1(f) of the image may be obtainedafter fitting is performed on c by using (f1, T1) and (f2, T2). When thefatigue state parameter of the observation object is the minimum fatiguethreshold f1, the image recognition device sets a presentation timecorresponding to the image to the minimum presentation time thresholdT1. When the fatigue state parameter of the observation object is themaximum fatigue threshold f2, the image recognition device sets apresentation time corresponding to the image to the maximum presentationtime threshold T2. A fitting relationship between T1(f) and f may ben-order linear fitting:

$\begin{matrix}{{T(f)} = {T_{1} + {\left( {f - f_{1}} \right)^{n} \cdot \frac{T_{2} - T_{1}}{\left( {f_{2} - f_{1}} \right)^{n}}}}} & \left( {1\text{-}9} \right)\end{matrix}$

It may be understood that this example is merely used to explain thisembodiment of this application without constituting any limitation. Thefitting between T1(f) and f is not limited to linear fitting, and mayalso be nonlinear fitting, for example, least square fitting.

In an embodiment, if the relationship that is between T(f) and f andthat is obtained through fitting according to the formula (1-8) is notdetermined by using the fatigue threshold and the presentation timethreshold, after the fatigue state parameter f0 corresponding to thetarget image is input to T(f), and the presentation time T0corresponding to the target image is obtained, the fatigue threshold andthe presentation time threshold may further be used for normalization.Specifically, a maximum presentation time obtained through calculationaccording to the formula (1-8) may be correspondingly set to the maximumpresentation time threshold T2. In other words, if the maximumpresentation time is obtained through calculation according to theformula (1-8), the presentation time of a corresponding image is set toT2. A minimum presentation time obtained through calculation accordingto the formula (1-8) may be correspondingly set to the minimumpresentation time threshold T1. In other words, if the minimumpresentation time is obtained through calculation according to theformula (1-8), the presentation time of a corresponding image is set toT1. According to the fatigue state parameter f0 of the target image andthe presentation time T0 corresponding to the target image, T0 may benormalized linearly and proportionally, to obtain a presentation timeT0′ of the target image obtained after T0 is normalized according to T1and T2.

It may be understood that the presentation time, calculated by using theformula (1-2), of the target image may be normalized in the same mannerby using the probability threshold and the presentation time threshold.The presentation time, calculated by using the formula (1-10), of thetarget image may be normalized in the same manner by using theprobability threshold, the fatigue threshold, and the presentation timethreshold.

When image recognition is performed only by using anelectroencephalogram, a presentation time of an image is set based on afatigue state parameter of the observation object. When the fatiguestate parameter of the observation object is larger, a longerpresentation time of the image is set by the image recognition device.This can reduce image miss detection caused by fatigue of theobservation object, and a miss detection rate can be reduced.

In an embodiment, after the presentation time of each image in the imagesequence is determined based on the fitting relationship between f andT, the presentation time of the corresponding image may be furtheradjusted based on the first recognition probability of each image. Whena first recognition probability is larger, a presentation time of theimage i may be adjusted to be longer. An adjusted presentation timeT′(c, f) may be determined based on a fitting relationship between thepresentation time of the image and t. A relationship between T′(c, f),and f and c may be obtained through n-order linear fitting, or may beobtained through non-linear fitting. This is not limited in thisembodiment of this application. A value of n may be a positive integergreater than 0. The image i may be any image in the image sequence. Aspecific fitting process may be adding an increment T1(c) to T(f)determined in the formula (1-8) or (1-9), so as to obtain a presentationtime T′(f, c) considering the impact of the fatigue state parameter.Alternatively, T(f) may be determined by using a mapping table, and anincrement T1(c) is added. A specific determining process may be similarto a process of determining the increment T1(f) when the duration impactparameter includes the first recognition probability c, and details arenot described herein again.

(3) The duration impact parameter includes the first recognitionprobability c and the fatigue state parameter f.

In this embodiment of this application, a presentation time T of animage in the image sequence may be determined in a manner of a firstmapping table based on a first recognition probability c of the imageand a fatigue state parameter f of the observation object. Specifically,the first mapping table may include a plurality of probabilities and aplurality of fatigue state parameters. One probability in the pluralityof probabilities and one fatigue state parameter in the plurality offatigue state parameters uniquely determine one presentation time.

The first recognition probability in the first mapping table may be aprobability value, or may be a probability interval. The fatigue stateparameter in the first mapping table may be a fatigue state parametervalue, or may be a fatigue state parameter interval. When the firstmapping table includes probability intervals and fatigue state parameterintervals, a presentation time corresponding to a first recognitionprobability is a presentation time corresponding to a probabilityinterval to which the first recognition probability belongs and aprobability interval to which the fatigue state parameter of theobservation object belongs. In this embodiment of this application, acase in which the first mapping table includes probability intervals andfatigue state parameter intervals is used as an example for description.It may be understood that, the first mapping table may further includeprobability intervals and fatigue values, or the first mapping table mayinclude probability values and fatigue intervals, or the first mappingtable may include probability values and fatigue values. This is notdescribed in detail. Table 6 is an example of another first mappingtable provided in this embodiment of this application.

TABLE 6 Example of another first mapping table c f (0.1, 0.3] (0.3, 0.5](0.5, 0.7] (0.7, 0.9] (1.15, 125] 0.4 s 0.3 s 0.2 s 0.1 s (1.25, 1.35]0.5 s 0.4 s 0.3 s 0.2 s (1.35, 1.45] 0.6 s 0.5 s 0.4 s 0.3 s

As shown in Table 6, when the first recognition probability falls withinan interval (0.1, 0.3], (0.3, 0.5], (0.5, 0.7], or (0.7, 0.9], and thefatigue state parameter of the observation object falls within aninterval (1.15, 1.25], (1.25, 1.35], or (1.35, 1.45]), a presentationtime may be determined by searching the foregoing first mapping table.For example, if the first recognition probability is 0.6, and thefatigue state parameter of the observation object is 1.28, the firstrecognition probability 0.6 falls within the probability interval (0.5,0.7], and the fatigue state parameter 1.28 of the observation objectfalls within the fatigue state parameter interval (1.25, 1.35]. Apresentation time corresponding to the probability interval (0.5, 0.7]and the fatigue state parameter interval (1.25, 1.35]) in the firstmapping table is 0.3 s. Therefore, the presentation time of the image isset to 0.3 s. It may be learned from the first mapping table that, whena first recognition probability is smaller, a longer image presentationtime is set, and when a fatigue state parameter is smaller, a longerimage presentation time is set.

T(c, f) may be obtained after n-order fitting is performed on the firstrecognition probability c and m-order linear fitting is performed on thefatigue state parameter f:

$\begin{matrix}{{T\left( {c,f} \right)} = {\sum\limits_{t = 0}^{n}{\sum\limits_{k = 0}^{m}{a_{t,k}c^{t}f^{k}}}}} & \left( {1\text{-}10} \right)\end{matrix}$

where T(c, f) is the presentation time, c is the first recognitionprobability, f is the fatigue state parameter, m is an order at whichT(c, f) fits f, n is an order at which T(c, f) fits c, both n and m arepositive integers greater than 0, t is an integer satisfying −n≤t≤n, kis an integer satisfying −m≤k≤m, c is a real number satisfying 0≤c≤1,and a_(t,k) is a coefficient of c^(t)f^(k).

Fitting between T(c, f) and the first recognition probability c and thefatigue state parameter f may also be non-linear fitting, for example,least square fitting. This is not limited in this application.

In an embodiment, when a magnitude difference between the twoindependent variables c and f is relatively large, impact of anindependent variable with a smaller magnitude in the fittingrelationship of the formula (1-10) is weakened by an independentvariable with a smaller magnitude. To reduce occurrence of the foregoingcase, when fitting is performed, the fatigue state parameter f may benormalized, for example, f is a value between [0, 1].

In an embodiment, T(c, f) may be obtained through fitting based on aprobability threshold, a fatigue threshold, and a presentation timethreshold. Specifically, T(c, f) may be obtained after linear fitting isperformed on c and f by using (c1, T2), (c2, T1), (f1, T1), and (f2,T2). T1 is a minimum presentation time threshold, T2 is a maximumpresentation time threshold, c1 is a minimum probability threshold of arecognition probability determined by using the computer visionalgorithm, c2 is a maximum probability threshold of a recognitionprobability determined by using the computer vision algorithm, f1 is aminimum fatigue threshold, and f2 is a maximum fatigue threshold.

For explanations of the probability threshold, the fatigue threshold,and the presentation time threshold, refer to the foregoing description,and details are not described herein again. Fitting between T(c, f), andc and f may also be non-linear fitting, for example, least squarefitting. This is not limited in this embodiment of this application.

2. Setting a Presentation Time of Each Image in the Image Sequence inReal Time or in Advance

A duration impact parameter of any image i in the image sequence may beobtained in real time before the image i is displayed in a process inwhich the image recognition device controls to display the imagesequence, or may be obtained in advance before the image recognitiondevice controls to start displaying the image sequence. Therefore, thepresentation time of the image i may be determined based on the durationimpact parameter in real time before the image i is displayed in theprocess of controlling to display the image sequence, or may bedetermined based on the duration impact parameter in advance before theimage sequence is controlled to be displayed. In addition, beforecontrolling to start displaying the image sequence, the imagerecognition device may further determine, based on a parameter in theduration impact parameter, the presentation time sequence correspondingto the image sequence, and then in a process of controlling to displaythe image sequence, adjust, in real time, the presentation time of theimage in the image sequence based on the other parameter in the durationimpact parameter. Descriptions are separately provided in the following:

(1) Determining a Presentation Time of any Image in the Image Sequencein Real Time

Descriptions are separately provided based on a different parameterincluded in the duration impact parameter.

If the duration impact parameter includes the fatigue state parameter,in a process of controlling to display the image sequence, for example,before any image i in the image sequence is controlled to be displayed,a fatigue state parameter corresponding to the image i may be obtained.A specific obtaining manner may be obtaining by detecting an observationobject by using a sensor. For example, an electroencephalogram signal ofthe observation object before the image i is displayed is detected, anda fatigue state parameter of the observation object before the image iis displayed is determined by analyzing the electroencephalogram signal,and is used as the fatigue state parameter corresponding to the image i.The detected fatigue state parameter before the image i is displayed maybe a fatigue state parameter of the observation object detected when oneor more images displayed before the image i are displayed, or may be afatigue state parameter obtained after a plurality of times ofmeasurement and averaging. Then, the presentation time of the image i isset based on the determined fatigue state parameter corresponding to theimage i.

In addition, the fatigue state parameter corresponding to the image imay also be obtained through prediction according to a fatigue rule. Forpredicting the fatigue state parameter corresponding to the imageaccording to the fatigue rule, refer to the foregoing embodiment.Details are not described herein again.

If the duration impact parameter includes the first recognitionprobability, in a process of controlling to display the image sequence,for example, before the image i is controlled to be displayed, a firstrecognition probability corresponding to the image i may be determinedby using a computer vision algorithm. Then, the presentation time of theimage i is set based on the determined first recognition probabilitycorresponding to the image i.

If the duration impact parameter includes the first recognitionprobability and the fatigue state parameter, in a process of controllingto display the image sequence, for example, before the image i iscontrolled to be displayed, a fatigue state parameter corresponding tothe image i may be obtained, where a specific obtaining manner may beobtaining by detecting an observation object by using a sensor, or maybe obtaining through prediction according to a fatigue rule; and beforethe image i is controlled to be displayed, a first recognitionprobability corresponding to the image i is determined by using acomputer vision algorithm. Then, the presentation time of the image i isset based on the first recognition probability and the fatigue stateparameter corresponding to the image i.

(2) Determining the Presentation Time Sequence Corresponding to theImage Sequence in Advance Before Controlling to Start Displaying theImage Sequence

Descriptions are separately provided based on a different parameterincluded in the duration impact parameter.

If the duration impact parameter includes the fatigue state parameter,before the image sequence is controlled to be displayed, a fatigue stateparameter corresponding to each image in the image sequence may beobtained. The obtained fatigue state parameter corresponding to eachimage in the image sequence may be obtained through prediction accordingto a fatigue rule. For predicting the fatigue state parametercorresponding to each image in the image sequence according to thefatigue rule, refer to the foregoing embodiment, and details are notdescribed herein again. Then, the presentation time corresponding toeach image in the image sequence may be determined based on the obtainedfatigue state parameter corresponding to each image in the imagesequence, so as to obtain the presentation time sequence correspondingto the image sequence.

If the duration impact parameter includes the first recognitionprobability, before the image sequence is controlled to be displayed,the image sequence may be processed by using a computer visionalgorithm, and a first recognition probability corresponding to eachimage is obtained through calculation by using the computer visionalgorithm. Then, the image recognition device may determine thepresentation time corresponding to each image in the image sequencebased on the first recognition probability, so as to obtain thepresentation time sequence corresponding to the image sequence. Afterthe presentation time sequence corresponding to the image sequence isobtained, the image sequence may be sequentially displayed under controlaccording to the presentation time sequence.

If the duration impact parameter includes the first recognitionprobability and the fatigue state parameter, before the image sequenceis controlled to be displayed, the image sequence may be processed byusing a computer vision algorithm, and a first recognition probabilitycorresponding to each image is obtained through calculation by using thecomputer vision algorithm; and a fatigue state parameter correspondingto each image in the image sequence may be predicted according to afatigue rule. Then, the image recognition device may determine thepresentation time corresponding to each image in the image sequencebased on the corresponding first recognition probability and fatiguestate parameter, so as to obtain the presentation time sequencecorresponding to the image sequence. After the presentation timesequence corresponding to the image sequence is obtained, the imagesequence may be sequentially displayed under control according to thepresentation time sequence. For example, before the image sequence iscontrolled to be displayed, for any image i in the image sequence, theimage recognition device may determine, by using the computer visionalgorithm, a first recognition probability corresponding to the image i,and then predict, by using the fatigue rule, a fatigue state parametercorresponding to the image i when the observation object observes theimage i. Then, the image recognition device determines a presentationtime of the image i based on the first recognition probability and thefatigue state parameter corresponding to the image i. The imagerecognition device may obtain the presentation time sequencecorresponding to the image sequence based on the presentation time ofeach image in the image sequence.

For a method for determining a presentation time of an image based onone or two parameters of a duration impact parameter corresponding tothe image, refer to the foregoing specific description of determiningthe presentation time sequence. Details are not described herein again.

(3) Determining the Presentation Time Sequence Corresponding to theImage Sequence in Advance Before Controlling to Start Displaying theImage Sequence, and Adjusting a Presentation Time of an Image in aProcess of Controlling to Display the Image Sequence

Specifically, the duration impact parameter may include the firstrecognition probability. In this case, before the image sequence iscontrolled to be displayed, the image recognition device processes theimage sequence by using a computer vision algorithm, so as to obtain,through calculation, a first recognition probability corresponding toeach image by using the computer vision algorithm. Then, the imagerecognition device may determine the presentation time corresponding toeach image in the image sequence based on the first recognitionprobability, so as to obtain the presentation time sequencecorresponding to the image sequence. After the presentation timesequence corresponding to the image sequence is obtained, the imagesequence may be sequentially displayed under control according to thepresentation time sequence.

In a process of controlling to display the image sequence, beforecontrolling to display any image i in the image sequence, the imagerecognition device may obtain a fatigue state parameter corresponding tothe image i. The fatigue state parameter corresponding to the image imay be obtained by detecting an observation object by using a sensor, ormay be obtained through prediction according to a fatigue rule. Apreviously determined presentation time of the image i is adjusted basedon the fatigue state parameter of the image i, so as to obtain anadjusted presentation time of the image i. When the image i iscontrolled to be displayed, the image i is displayed according to theadjusted presentation time of the image i. For any image in the imagesequence, the foregoing process of adjusting a presentation time of theimage by using a fatigue state parameter may be executed. A policy ofadjusting a presentation time of an image may be as follows: When afatigue state parameter is larger, the presentation time of the image isincreased by a greater amplitude; and when a fatigue state parameter issmaller, the presentation time of the image is decreased by a greateramplitude. For specific implementation of adjusting a presentation timeof an image, refer to the foregoing embodiment. Details are notdescribed herein again.

3. Multi-Observation Object Recognition, Multi-Round Recognition, and aBrain-Computer Fusion Weight

(1) Multi-Observation Object Recognition

In an embodiment, there are at least two observation objects, thefeedback signals are at least two feedback signals respectivelygenerated when the at least two observation objects observe the imagesequence, and the target recognition probability is determined based onthe computer vision signal and the at least two feedback signals.

Specifically, for each observation object, the image recognition devicemay fuse the computer vision signal obtained through calculation byusing the computer vision algorithm and a feedback signal of theobservation object, to obtain a brain-computer combination imagerecognition result. In other words, at least two brain-computercombination image recognition results are obtained, and the imagerecognition result is whether the image includes the preset imagefeature. The image recognition device determines a final imagerecognition result based on the at least two recognition results, forexample, may perform weighted summation on the at least two recognitionresults to determine the final image recognition result.

For example, the observation objects are A, B, and C, and brain-computercombination image recognition is performed on any image i in the imagesequence. In this case, a brain-computer combination image recognitionresult corresponding to A may be calculated based on the computer visionsignal and a feedback signal of A. The brain-computer combination imagerecognition result corresponding to A is a probability a that the imagei includes the preset image feature. The same method may be used toseparately calculate a brain-computer combination image recognitionresult corresponding to B and a brain-computer combination imagerecognition result corresponding to C. The brain-computer combinationimage recognition result corresponding to B is a probability b that theimage i includes the preset image feature. The brain-computercombination image recognition result corresponding to C is a probabilityc that the image i includes the preset image feature. Then a final imagerecognition result is determined in a weighted summation manner based onweights Wa, Wb, and Wc respectively used by the brain-computercombination image recognition results corresponding to A, B, and C, thatis, a probability that the image i includes the preset image feature isWa×a+Wb×b+Wc×c.

Alternatively, the image recognition device may first obtain an overallfeedback signal feature through calculation based on the at least twofeedback signals by using a specific policy, for example, perform tensorsplicing on feedback signal features of the at least two feedbacksignals to obtain the overall feedback signal feature. Then the imagerecognition device fuses the overall feedback signal feature and anoutput result of the computer vision algorithm to obtain the final imagerecognition result. In this embodiment of this application, the fusionof the feedback signal and the computer vision signal obtained throughcalculation by using the computer vision algorithm may be probabilityfusion, or may be feature fusion. For a specific process, refer to theforegoing specific description of brain-computer combination imagerecognition. Details are not described herein again.

For example, the observation objects are A, B, and C, and brain-computercombination image recognition is performed on any image i in the imagesequence. A feedback signal obtained when A observes the image i, afeedback signal obtained when B observes the image i, and a feedbacksignal obtained when C observes the image i are fused to obtain anoverall feedback signal. A fusion process may be signal superposition,or may be feature fusion, for example, may be specifically splicing oftensor features. This is not limited in this application. The overallfeedback signal includes brain recognition results of A, B, and C forthe image i. Then the overall feedback signal is fused with the computervision signal to obtain a target recognition signal. For a process inwhich the feedback signal and the computer vision signal are fused, andthe tensor features are spliced to perform feature fusion, refer to theforegoing embodiment. Details are not described herein again.

It may be understood that the foregoing examples for brain-computercombination image recognition with a plurality of observation objectsare only used to describe this embodiment of this application, andshould not constitute a limitation.

In an implementation, when there are at least two observation objects,the image recognition device may determine impact of fatigue stateparameters of the at least two observation objects on a presentationtime of any image i in the image sequence. There are at least twofatigue state parameters corresponding to the image i. The imagerecognition device may determine a weighted sum of the fatigue stateparameters of the at least two observation objects, and then determinethe presentation time of the image i based on the weighted sum of thefatigue state parameters. The presentation time of the image i may bepositively correlated with the weighted sum of the fatigue stateparameters. A fatigue state parameter weight of each observation objectmay be determined by the image recognition device based on a statisticalfeature of a fatigue state parameter of each observation object.

In addition, when there are at least two observation objects, the imagerecognition device may first determine an intermediate value of fatiguestate parameters of the at least two observation objects correspondingto the image i, and then determine the presentation time of the image ibased on the intermediate value of the fatigue state parameters. Thepresentation time of image i may be positively correlated with theintermediate value of the fatigue state parameters.

It may be understood that this example is merely used to explain thisembodiment of this application without constituting any limitation. Theimage recognition device may determine any statistical value, forexample, an average, of the fatigue state parameters of the at least twoobservation objects, and then determine a presentation time of an imagebased on the statistical value of the fatigue state parameters.

A plurality of observation objects simultaneously perform brain-computercombination image recognition on an image in the image sequence. Thiscan reduce a random error caused by a subjective reason of anobservation object in a case of one observation object, therebyimproving accuracy of brain-computer combination image recognition.

(2) Multi-Round Recognition

In an implementation, the image recognition device may select imageswhose target recognition probabilities are between a second probabilitythreshold and a third probability threshold in the image sequence. Thesecond probability threshold is a threshold used by the imagerecognition device to determine that an image does not include thepreset image feature. When a probability that an image includes thepreset image feature is less than or equal to the second probabilitythreshold, the image recognition device determines that the image is notan image that includes the preset image feature. The third probabilitythreshold is a threshold used by the image recognition device todetermine that an image includes the preset image feature. When aprobability that an image includes the preset image feature is greaterthan or equal to the third probability threshold, the image recognitiondevice determines that the image is an image that includes the presetimage feature. The second probability threshold is greater than or equalto the third probability threshold. The selected images are used as anew image sequence for brain-computer combination image recognition.

Because the new image sequence comes from the initial image sequencebefore selection, and the presentation time of each image in the initialimage sequence may be determined according to operation S401 in theimage recognition method described in FIG. 4, a presentation time ofeach image in the new image sequence on the display device may be thedetermined presentation time of the image in the initial image sequence,or may be re-determined. This is not limited in this application.Re-determining the presentation time of each image in the new imagesequence on the display device may also be adding an increment on thebasis of the presentation time of the image in the initial imagesequence. If the image recognition device sets, for any image i in theimage sequence, a determined presentation time of the image i in theinitial image sequence to Ts, in the new image sequence, the imagerecognition device may set a presentation time of the image i to Ts+Tm,or may set the presentation time of the image i to Ts×(1+s %), or mayset the presentation time of the image i to Ts×t. Tm indicates aduration increment, s indicates a duration increment percentage, and tindicates a duration increment multiple. Tm, s, and t may be constantsor variables.

For example, the second probability threshold is set to 30%, and thethird probability threshold is set to 70%. The image recognition devicemay determine that the image i includes the preset image feature whendetecting that a second probability of the image i is greater than orequal to 70%. The image recognition device may determine that the imagei does not include the preset image feature when detecting that thesecond probability of the image i is less than or equal to 30%. In thiscase, the image recognition device may select images whose secondprobabilities are between 30% and 70% in the image sequence, and use theimages as a new image sequence for brain-computer combination imagerecognition. For example, in the original image sequence, thepresentation time of the image i determined by the image recognitiondevice according to operation S401 in the image recognition methoddescribed in FIG. 4 is 200 ms, and the second probability of the image ithat is obtained by the image recognition device through brain-computercombination calculation is 52%, which is between 30% and 70%. In otherwords, the target recognition probability 52% of the image i fallswithin a range of [30%, 70%]. In the new image sequence, the imagerecognition device may still set the presentation time of the image i to200 ms, or may set the presentation time of the image i to (200+100) ms,or may set the presentation time of the image i to 200×(1+20%) ms, ormay set the presentation time of the image i to 200×2 ms. It may beunderstood that this example is merely used to explain this embodimentof this application without constituting any limitation.

In addition, the second probability threshold and the third probabilitythreshold may be determined based on a sensitivity requirement of theimage recognition system. The second probability threshold mayalternatively be set to be greater than a probability threshold of animage used to indicate that the image does not include the preset imagefeature. Similarly, the third probability threshold may alternatively beset to be less than a probability threshold of an image used to indicatethat the image includes the preset image feature. For example, the imagerecognition device determines that a probability threshold of an imageindicating that the image does not include the preset image feature is20%, that is, when a probability that the image includes the presetimage feature is less than or equal to 20%, the image recognition devicedetermines that the image is not an image that includes the preset imagefeature. The image recognition device determines that a probabilitythreshold of an image indicating that the image includes the presetimage feature is 80%, that is, when a probability that the imageincludes the preset image feature is greater than or equal to 80%, theimage recognition device determines that the image is an image thatincludes the preset image feature. In this case, the second probabilitythreshold is set to 30%, and the third probability threshold is set to70%. In other words, images whose second probabilities are between 30%and 70% in the image sequence are selected as a new image sequence forbrain-computer combination image recognition.

If the image recognition device re-detects that a second probability ofan image in a new image sequence is still between the second probabilitythreshold and the third probability threshold, the image recognitiondevice may determine that the image is not an image that includes thepreset image feature, and the image may be put into a new round of imagesequence again, and then iterated to the image recognition system forre-recognition. A quantity of rounds of iterative detection in thisembodiment of this application is not limited, and may be determinedbased on a sensitivity requirement of the image recognition device.

It may be understood that the foregoing example is merely used toexplain this embodiment of this application without constituting anylimitation.

Images, in the image sequence, with relatively great uncertainty ofwhether the target image feature is included are selected, for aplurality of times, as a new image sequence for brain-computercombination image recognition. In this way, a suspicious object in theimage sequence can be filtered out, a probability of misjudgment by theimage recognition device is reduced, and accuracy of brain-computercombination image recognition can be improved.

(3) Brain-Computer Fusion Weight

In an implementation, for each image in the image sequence, when thefeedback signal and the computer vision signal are fused, the feedbacksignal and the computer vision signal may be fused based on weights. Afusion weight (namely, a first weight) used by the feedback signal maybe related to at least one of the first recognition probability, thefatigue state parameter of the observation object, and the presentationtime. The first weight is inversely correlated with the firstrecognition probability, the first weight is inversely correlated withthe fatigue state parameter, and the first weight is positivelycorrelated with a presentation time of an image.

First, when a first recognition probability of recognition by using thecomputer vision algorithm is larger, it indicates that a probabilitythat the image includes the preset image feature is higher, that is, arecognition accuracy rate of the computer vision algorithm recognitionis higher. Therefore, a fusion weight used by recognition by using thecomputer vision algorithm may be increased, and a fusion weight used bybrain recognition of an observation object may be decreased, so as toreduce a miss detection rate. Second, if a fatigue state parameter ofthe observation object is larger, it indicates that efficiency of brainrecognition of the observation object in a fatigue state is lower andthe miss detection rate is higher. Therefore, when the fatigue stateparameter of the observation object is larger, the weight of brainrecognition of the observation object is smaller. Finally, a longerpresentation time of an image indicates a longer time for observing bythe observation object. Therefore, when accuracy of brain recognition ofthe observation object is higher, a miss detection rate is lower, andthe weight of recognition by using the feedback signal may be set to belarger. In other words, a longer presentation time of an image indicatesa larger fusion weight of the feedback signal.

Based on at least one of the first recognition probability, the fatiguestate parameter of the observation object, and the presentation time ofthe image, the first weight may be determined in a manner of a mappingtable, or may be determined in a manner of function fitting. Thefunction fitting may be n-order linear fitting, nonlinear fitting, orthe like. This is not limited in this embodiment of this application.

In addition, in this embodiment of this application, the obtained imagesequence and the corresponding presentation time sequence are used tocontrol display of the image sequence, and perform brain-computercombination image recognition. It may be understood that the obtainedimage sequence and the corresponding presentation time sequence are notlimited to the foregoing application scenario, and the obtained imagesequence and the corresponding presentation time sequence may be outputor stored. With evolution of an image-related system and emergence of anew service scenario, the technical solutions provided in thisapplication are also applicable to similar technical problems.

FIG. 6 is a schematic flowchart of an image presentation time adjustmentmethod according to an embodiment of this application. In this method,an image presentation time adjustment device adjusts a presentation timeof an image based on a duration impact parameter. As shown in FIG. 6,the image presentation time adjustment method includes but is notlimited to the following operations S201 to S203.

S201. The image presentation time adjustment device obtains an imagesequence based on RSVP.

S202. The image presentation time adjustment device adjusts apresentation time corresponding to each image in the image sequencebased on a corresponding duration impact parameter for each image in theimage sequence.

A first recognition probability is inversely correlated with apresentation time, and a fatigue state parameter is positivelycorrelated with a presentation time.

S203. The image presentation time adjustment device controls display ofthe image sequence based on an adjusted presentation time correspondingto each image in the image sequence.

In this embodiment of this application, before the presentation timecorresponding to each image in the image sequence is adjusted based on acorresponding duration impact parameter for each image in the imagesequence, presentation times of all images in the image sequence may beequal or unequal. For descriptions of the fatigue state parameter, thefatigue rule, the first recognition probability, and the feedback signalin this embodiment of this application, refer to the embodimentdescribed in FIG. 4. Details are not described herein again.

In an embodiment, the image sequence that is displayed by the imagepresentation time adjustment device based on the adjusted presentationtime may be used for brain-computer combination image recognition. For aspecific description of brain-computer combination image recognitionbased on an image sequence, refer to the embodiment described in FIG. 4.Details are not described herein again. When brain-computer combinationimage recognition is performed, the image presentation time adjustmentdevice may adjust a specified presentation time sequence based on aduration impact parameter. Attention resources of the brain of theobservation object in a time dimension can be better used to recognizean image, and more attention resources of the observation object in thetime dimension are allocated to an image with relatively greatuncertainty. Therefore, a miss detection rate of image recognition canbe reduced, and efficiency of brain-computer collaboration imagerecognition is improved.

In an embodiment, the image presentation time adjustment device mayadjust the presentation time corresponding to each image in the imagesequence based on the corresponding duration impact parameter for eachimage in the image sequence before controlling to start displaying theimage sequence, so as to obtain an adjusted presentation time sequence.Then, the image sequence is controlled to be sequentially displayedbased on the adjusted presentation time sequence.

In an embodiment, in a process of controlling to display the imagesequence, before controlling to display any image i in the imagesequence, the image presentation time adjustment device may obtain, inreal time, a duration impact parameter of the image i, and adjust apresentation time of the image i based on the duration impact parameterof the image i. Then, display of the image i is controlled based on theadjusted presentation time of the image i.

In an embodiment, the image presentation time adjustment device mayfirst adjust the presentation time corresponding to each image in theimage sequence based on the corresponding duration impact parameter foreach image in the image sequence before controlling to start displayingthe image sequence, so as to obtain an adjusted presentation timesequence. In a process of controlling to display the image sequence,before controlling to display any image i in the image sequence, theimage presentation time adjustment device obtains, in real time, anotherduration impact parameter of the image i, and adjusts the presentationtime of the image i again based on the another duration impact parameterof the image i. Then, display of the image i is controlled based on theadjusted presentation time of the image i.

Specific descriptions of the foregoing three adjustment manners may besimilar to specific descriptions of manners for determining thepresentation time sequence corresponding to the image sequence in theembodiment described in FIG. 4, and details are not described hereinagain.

In an embodiment, the image sequence may also include N images selectedfrom M images received from a camera device.

In an embodiment, that the image presentation time adjustment deviceadjusts a presentation time corresponding to each image in the imagesequence based on a corresponding duration impact parameter for eachimage in the image sequence may specifically include: For any image j inthe image sequence, the image presentation time adjustment device mayfirst determine a presentation time offset based on a duration impactparameter, and then adjust a presentation time of the image j based onthe determined presentation time offset. The presentation time offsetdetermined based on the duration impact parameter may be determined byusing a mapping table, or may be determined by using a fitting formula.A process of determining the presentation time offset by using a mappingtable is specifically as follows: For the image j, the presentation timeoffset of the image j is found from a third mapping table based on theduration impact parameter of the image j, where the third mapping tableincludes a plurality of duration impact parameters and presentation timeoffsets respectively corresponding to the plurality of duration impactparameters. For a specific description of determining the presentationtime offset by using a mapping table, refer to the description ofdetermining the presentation time according to the first mapping tablein the embodiment described in FIG. 4. In other words, the third mappingtable may be similar to the first mapping table, and details are notdescribed herein again.

A process of determining the presentation time offset by using a fittingformula is specifically as follows: When the duration impact parameterincludes the first recognition probability, that the image presentationtime adjustment device adjusts a presentation time corresponding to eachimage in the image sequence based on a corresponding duration impactparameter for each image in the image sequence includes: The imagepresentation time adjustment device obtains the presentation time offsetof each image in the image sequence by using the following fittingformula:

$\begin{matrix}{{\Delta{T(c)}} = {\sum\limits_{t = 0}^{n}{a_{j}c^{t}}}} & \left( {1\text{-}11} \right)\end{matrix}$

where ΔT(c) is the presentation time offset, c is the first recognitionprobability, c is a real number satisfying 0≤c≤1, n is an order at whichΔT(c) fits c, n is an integer greater than 0, t is an integer satisfying−n≤t≤n, and a_(t) is a coefficient of c^(t).

In an embodiment, ΔT(c) is obtained after n-order linear fitting isperformed on c by using (c1, T2) and (c2, T1). T1 is a minimumpresentation time threshold, T2 is a maximum presentation timethreshold, c1 is a minimum probability threshold of a recognitionprobability determined by using the computer vision algorithm, and c2 isa maximum probability threshold of a recognition probability determinedby using the computer vision algorithm. For descriptions of the minimumpresentation time threshold, the maximum presentation time threshold,the minimum probability threshold, and the maximum probabilitythreshold, refer to the embodiment described in FIG. 4. Details are notdescribed herein again.

Specifically, for example, if the presentation time of each image in theimage sequence is Tc before adjustment, the n-order linear fitting maybe as follows:

$\begin{matrix}{{\Delta{T(c)}} = {T_{1} - {Tc} + {\left( {c - {c1}} \right)^{n} \cdot \frac{\left( {{T1} - {Tc}} \right) - \left( {{T2} - {Tc}} \right)}{\left( {{c2} - {c1}} \right)^{n}}}}} & \left( {1\text{-}12} \right)\end{matrix}$

In an embodiment, when a first recognition probability of an image q isgreater than or equal to c2, the first recognition probability is usedto determine that the image q includes the preset image feature. Whenthe first recognition probability of the image q is less than or equalto c1, the first recognition probability is used to determine that theimage q does not include the preset image feature, where the image q isany image in the image sequence.

In an embodiment, the duration impact parameter includes the fatiguestate parameter, and that the image presentation time adjustment deviceadjusts a presentation time corresponding to each image in the imagesequence based on a corresponding duration impact parameter for eachimage in the image sequence includes: The image presentation timeadjustment device obtains the presentation time offset of each image inthe image sequence by using the following fitting formula:

$\begin{matrix}{{\Delta{T(f)}} = {\sum\limits_{k = 0}^{m}{a_{k}f^{k}}}} & \left( {1\text{-}13} \right)\end{matrix}$

where ΔT(f) is the presentation time offset, f is the fatigue stateparameter, m is an order at which ΔT(f) fits f, m is a positive integergreater than 0, k is an integer satisfying −m≤k≤m, and a_(k) is acoefficient of f^(k).

In an embodiment, ΔT(f) is obtained after n-order linear fitting isperformed on f by using (f1, T1) and (f2, T2). T1 is a minimumpresentation time threshold, T2 is a maximum presentation timethreshold, f1 is a minimum fatigue threshold, and f2 is a maximumfatigue threshold. For descriptions of the minimum fatigue threshold andthe maximum fatigue threshold, refer to the embodiment described in FIG.4. Details are not described herein again.

Specifically, for example, if the presentation time of each image in theimage sequence is Tc before adjustment, the n-order linear fitting maybe as follows:

$\begin{matrix}{{\Delta{T(f)}} = {T_{1} - {Tc} + {\left( {f - {f1}} \right)^{n} \cdot \frac{\left( {{f2} - {Tc}} \right) - \left( {{f1} - {Tc}} \right)}{\left( {{f2} - {f1}} \right)^{n}}}}} & \left( {1\text{-}14} \right)\end{matrix}$

In an embodiment, the duration impact parameter includes the firstrecognition probability and the fatigue state parameter, and that theimage presentation time adjustment device adjusts a presentation timecorresponding to each image in the image sequence based on acorresponding duration impact parameter for each image in the imagesequence includes: The image presentation time adjustment device obtainsthe presentation time offset of each image in the image sequence byusing the following fitting formula:

$\begin{matrix}{{\Delta{T\left( {c,f} \right)}} = {\sum\limits_{t = 0}^{n}{\sum\limits_{k = 0}^{m}{a_{t,k}c^{t}f^{k}}}}} & \left( {1\text{-}15} \right)\end{matrix}$

where ΔT(c, f) is the presentation time offset, c is the firstrecognition probability, f is the fatigue state parameter, m is an orderat which ΔT(c, f) fits f, n is an order at which ΔT(c, f) fits c, both nand m are positive integers greater than 0, t is an integer satisfying−n≤t≤n, k is an integer satisfying −m≤k≤m, c is a real number satisfying0≤c≤1, and a_(t,k) is a coefficient of c^(t)f^(k).

The presentation time corresponding to each image in the image sequenceis adjusted based on the corresponding presentation time offset of eachimage in the image sequence.

In an embodiment, ΔT(c, f) is obtained after linear fitting is performedon c and f by using (c1, T2), (c2, T1), (f1, T1), and (f2, T2). T1 is aminimum presentation time threshold, T2 is a maximum presentation timethreshold, c1 is a minimum probability threshold of a recognitionprobability determined by using the computer vision algorithm, c2 is amaximum probability threshold of a recognition probability determined byusing the computer vision algorithm, f1 is a minimum fatigue threshold,and f2 is a maximum fatigue threshold.

A specific description of determining the presentation time offset byusing the fitting formula based on the duration impact parameter may besimilar to the description of determining the presentation time by usingthe fitting formula in the embodiment described in FIG. 4.

In an embodiment, when it is detected that a corresponding fatigue stateparameter obtained when the observation object observes an image r isgreater than or equal to a first fatigue threshold, the imagepresentation time adjustment device may control to stop displayingimages to be displayed after the image r in the image sequence, andobtain an image whose corresponding first recognition probability isgreater than or equal to a first probability threshold in the images tobe displayed after the image r; and when it is detected that the fatiguestate parameter of the observation object is less than or equal to asecond fatigue threshold, the image presentation time adjustment devicecontrols to sequentially display the image whose first recognitionprobability is greater than or equal to the first probability thresholdin the images to be displayed after the image r, where the image r isany image in the image sequence. For description of controlling, basedon the fatigue state parameter, to stop displaying the image sequence,or to start displaying the image sequence again, refer to the embodimentdescribed in FIG. 4. Details are not described herein again.

In an embodiment, there are at least two observation objects, and thefatigue state parameter is at least two fatigue state parametersrespectively generated when the at least two observation objects observea same image. A presentation time of an image u is positively correlatedwith a weighted sum of the at least two fatigue state parameters, wherethe image u is any image in the image sequence. The fatigue stateparameter includes the at least two fatigue state parametersrespectively generated when the at least two observation objects observea same image. For a description of performing brain-computer combinationimage recognition on one image sequence by a plurality of observationobjects, refer to the embodiment described in FIG. 4. Details are notdescribed herein again.

In an embodiment, that the image recognition device fuses, for eachimage in the image sequence, a corresponding computer vision signal anda corresponding feedback signal to obtain a target recognition signal ofeach image in the image sequence includes: determining, for each imagein the image sequence based on at least one of the first recognitionprobability, the fatigue state parameter, and the presentation time, afirst weight corresponding to each image in the image sequence, wherethe first weight is a weight used when the corresponding feedback signalis used to determine the target recognition signal, the first weight isinversely correlated with the first recognition probability, the firstweight is inversely correlated with the fatigue state parameter, and thefirst weight is positively correlated with the presentation time; andfusing, for each image in the image sequence based on a correspondingfirst weight, a corresponding computer vision signal and a correspondingfeedback signal to obtain the target recognition signal of each image inthe image sequence. For descriptions of the first weight and the fusionof the computer vision signal and the feedback signal based on theweight, refer to the embodiment described in FIG. 4. Details are notdescribed herein again.

For a description of the fusion of the computer vision signal and thefeedback signal, refer to the foregoing description. Details are notdescribed herein again.

In an embodiment, the image presentation time adjustment device usesimages whose corresponding target recognition probabilities are betweena second probability threshold and a third probability threshold in theimage sequence as a new image sequence. The new image sequence may beused to re-execute a brain-computer combination image recognitionprocess. For specific descriptions of the second threshold and the thirdprobability threshold, refer to the specific descriptions of theembodiment described in FIG. 4. Details are not described herein again.

It may be understood that, in the embodiment described in FIG. 6, forexplanations of related descriptions, refer to the embodiment describedin FIG. 4. Anew embodiment may be obtained with reference to any one ormore implementations of the embodiment described in FIG. 4 and theembodiment described in FIG. 6. This is not limited in the embodimentsof this application.

The method in the embodiments of the present invention is describedabove in detail, and an apparatus in an embodiment of the presentinvention is provided below.

FIG. 7 is a schematic structural diagram of an image recognition device40 according to an embodiment of this application. As shown in FIG. 7,the device may include a setting unit 101, a calculation unit 102, anobtaining unit 103, and a fusion unit 104.

The setting unit 101 is configured to set a presentation time sequencecorresponding to an image sequence, where the image sequence includes Nimages, N is a positive integer, the presentation time sequence includesa presentation time of each image in the image sequence, a presentationtime of an image i is used to indicate a time period from a presentationstart moment of the image i to a presentation start moment of a nextadjacent image, the image i is any image in the image sequence, thepresentation time sequence includes at least two unequal presentationtimes, a difference between any two presentation times of the at leasttwo unequal presentation times is k×Δ, k is a positive integer, and Δ isa preset time period value.

The calculation unit 102 is configured to process the image sequence byusing a computer vision algorithm, to obtain a computer vision signalcorresponding to each image in the image sequence.

The obtaining unit 103 is configured to obtain a feedback signal that isgenerated when an observation object watches the image sequencedisplayed in the presentation time sequence and that corresponds to eachimage in the image sequence, where the feedback signal is used toindicate a reaction of the observation object to the watched image.

The fusion unit 104 is configured to fuse, for each image in the imagesequence, a corresponding computer vision signal and a correspondingfeedback signal to obtain a target recognition signal of each image inthe image sequence, where the target recognition signal is used forimage recognition.

In an embodiment, the image recognition device further includes areceiving unit 105 and a selection unit 106. The receiving unit 105 isconfigured to receive M images from a camera device, where M is aninteger greater than 1.

The selection unit 106 is configured to select N images from the Mimages as the image sequence, where N is less than or equal to M.

In an embodiment, the setting unit 101 is specifically configured todetermine a corresponding presentation time for each image in the imagesequence based on a duration impact parameter, to obtain thepresentation time sequence corresponding to the image sequence.

The duration impact parameter includes at least one of the firstrecognition probability and a fatigue state parameter, the firstrecognition probability is used to indicate a probability, obtained byusing the computer vision algorithm, that an image includes a presetimage feature, the fatigue state parameter is used to indicate a fatiguedegree of the observation object when the observation object observes animage, the presentation time is inversely correlated with the firstrecognition probability, and the presentation time is positivelycorrelated with the fatigue state parameter.

In an embodiment, the duration impact parameter includes the fatiguestate parameter, and the image recognition device 40 further includes aprediction unit 107. The prediction unit 107 is configured to predict,according to a fatigue rule, the fatigue state parameter correspondingto each image in the image sequence, where the fatigue rule is used toindicate a change rule of a fatigue degree of the observation object.

In an embodiment, the obtaining unit 103 is further configured to: in aprocess of displaying the image sequence in the presentation timesequence, obtain the fatigue state parameter corresponding to an imagej, and adjust, based on the fatigue state parameter corresponding to theimage j, a presentation time, in the presentation time sequence,corresponding to an image to be displayed after the image j in the imagesequence, where the image j is any image in the image sequence.

In an embodiment, the obtaining unit 103 is specifically configured toobtain the fatigue state parameter based on fatigue state informationthat is sent by a sensor and that is obtained when the observationobject watches the image j.

In an embodiment, the setting unit 101 is specifically configured to:for each image in the image sequence, find a presentation timecorresponding to the duration impact parameter from a first mappingtable, where the first mapping table includes a plurality of durationimpact parameters and presentation times respectively corresponding tothe plurality of duration impact parameters.

In an embodiment, the duration impact parameter includes the firstrecognition probability.

The setting unit 101 is specifically configured to obtain thepresentation time of each image in the image sequence by using thefollowing fitting formula:

${{T(c)} = {\sum\limits_{t = 0}^{n}{a_{j}c^{t}}}};$where

T(c) is the presentation time, c is the first recognition probability, cis a real number satisfying 0≤c≤1, n is an order at which T(c) fits c, nis an integer greater than 0, t is an integer satisfying −n≤t≤n, anda_(t) is a coefficient of c^(t).

In an embodiment, T(c) is obtained after n-order linear fitting isperformed on c by using (c1, T2) and (c2, T1). T1 is a minimumpresentation time threshold, T2 is a maximum presentation timethreshold, c1 is a minimum probability threshold of a recognitionprobability determined by using the computer vision algorithm, and c2 isa maximum probability threshold of a recognition probability determinedby using the computer vision algorithm.

In an embodiment, the duration impact parameter includes the fatiguestate parameter, and the setting unit 101 is specifically configured toobtain the presentation time of each image in the image sequence byusing the following fitting formula:

${{T(f)} = {\sum\limits_{k = 0}^{m}{a_{k}f^{k}}}};$where

T(f) is the presentation time, f is the fatigue state parameter, m is anorder at which T(f) fits f, m is a positive integer greater than 0, k isan integer satisfying −m≤k≤m, and a_(k) is a coefficient of f^(k).

In an embodiment, the duration impact parameter includes the firstrecognition probability and the fatigue state parameter, and the settingunit 101 is specifically configured to obtain the presentation time ofeach image in the image sequence by using the following fitting formula:

${{T\left( {c,f} \right)} = {\sum\limits_{t = 0}^{n}{\sum\limits_{k = 0}^{m}{a_{t,k}c^{t}f^{k}}}}};$where

T(c, f) is the presentation time, c is the first recognitionprobability, f is the fatigue state parameter, m is an order at whichT(c, f) fits f, n is an order at which T(c, f) fits c, both n and m arepositive integers greater than 0, t is an integer satisfying −n≤t≤n, kis an integer satisfying −m≤k≤m, c is a real number satisfying 0≤c≤1,and a_(t,k) is a coefficient of c^(t)f^(k).

In an embodiment, the image recognition device further includes adetection unit 108, configured to: when it is detected that acorresponding fatigue state parameter obtained when the observationobject observes an image q is greater than or equal to a first fatiguethreshold, control to stop displaying images to be displayed after theimage q in the image sequence.

The obtaining unit 103 is further configured to obtain an image whosecorresponding first recognition probability is greater than or equal toa first probability threshold in the images to be displayed after theimage q, where the image q is any image in the image sequence.

The detection unit 108 is further configured to: when it is detectedthat the fatigue state parameter of the observation object is less thanor equal to a second fatigue threshold, control to sequentially displaythe image whose first recognition probability is greater than or equalto the first probability threshold in the images to be displayed afterthe image q.

In an embodiment, there are at least two observation objects. The fusionunit 104 is specifically configured to fuse, for each image in the imagesequence, a corresponding computer vision signal and at least twocorresponding feedback signals to obtain a target recognition signal ofeach image in the image sequence.

In an embodiment, the fatigue state parameter includes at least twofatigue state parameters respectively generated when the at least twoobservation objects observe a same image.

In an embodiment, the fusion unit 104 is specifically configured to:determine, for each image in the image sequence based on at least one ofthe first recognition probability, the fatigue state parameter, and thepresentation time, a first weight corresponding to each image in theimage sequence, where the first weight is a weight used when thecorresponding feedback signal is used to determine the targetrecognition signal, the first weight is inversely correlated with thefirst recognition probability, the first weight is inversely correlatedwith the fatigue state parameter, and the first weight is positivelycorrelated with the presentation time.

The fusion unit 104 is further configured to fuse, for each image in theimage sequence based on a corresponding first weight, a correspondingcomputer vision signal and a corresponding feedback signal to obtain thetarget recognition signal of each image in the image sequence.

In an embodiment, the computer vision signal is a first recognitionprobability determined by using the computer vision algorithm.

The calculation unit 102 is further configured to calculate, for eachimage in the image sequence, a second recognition probability of eachimage in the image sequence based on a corresponding feedback signal,where the second recognition probability is used to indicate aprobability that the observation object determines that the imageincludes the preset image feature. The fusion unit 104 is specificallyconfigured to calculate, for each image in the image sequence, a targetrecognition probability of each image in the image sequence based on thecorresponding first recognition probability and the corresponding secondrecognition probability.

In an embodiment, the computer vision signal is an image featuredetermined by using the computer vision algorithm.

The calculation unit 102 is further configured to determine, for eachimage in the image sequence based on a corresponding feedback signal, afeedback signal feature corresponding to each image in the imagesequence. The fusion unit 104 is specifically configured to: perform,for each image in the image sequence, feature fusion on thecorresponding image feature and the corresponding feedback signalfeature, to obtain a fused feature corresponding to each image in theimage sequence; and determine, for each image in the image sequence, atarget recognition probability of each image in the image sequence basedon the corresponding fused feature.

In an embodiment, the device further includes a determining unit 109,configured to determine, from the image sequence based on the targetrecognition probability of each image in the image sequence, S images asimages including the preset image feature, where the target recognitionprobabilities of the S images meet a preset condition, and S is aninteger less than or equal to N.

In an embodiment, the setting unit 101 is further configured to useimages that are in the image sequence and whose corresponding targetrecognition probabilities are between a second probability threshold anda third probability threshold as a new image sequence.

In an embodiment, the feedback signal is an electroencephalogram signal.

It should be noted that, for implementation of each unit in the imagerecognition device 40, refer to corresponding descriptions in the methodembodiment shown in FIG. 4. Details are not described herein again. Theimage recognition device 40 may be the image recognition device 40 inthe system architecture shown in FIG. 1.

FIG. 8 is a schematic structural diagram of an image presentation timedetermining device 50 according to an embodiment of this application. Asshown in FIG. 8, the device 50 may include an obtaining unit 501, asetting unit 502, and an output and storage unit 503.

The obtaining unit 501 is configured to obtain a plurality of images,where the plurality of images may form an image sequence.

The setting unit 502 is configured to: set a corresponding presentationtime for each image in the plurality of images based on a durationimpact parameter, to obtain a presentation time sequence correspondingto the plurality of images. The duration impact parameter includes atleast one of a first recognition probability and a fatigue stateparameter, the presentation time is inversely correlated with the firstrecognition probability, and the presentation time is positivelycorrelated with the fatigue state parameter. The first recognitionprobability is used to indicate a probability, obtained by using acomputer vision algorithm, that an image includes a preset imagefeature. The fatigue state parameter is used to indicate a fatiguedegree of the observation object when the observation object observes animage. A presentation time of an image i is used to indicate a timeperiod from a presentation start moment of the image i to a presentationstart moment of a next adjacent image, and the image i is any image inthe plurality of images.

The output and storage unit 503 is configured to output or store theplurality of images and the presentation time sequence corresponding tothe plurality of images.

In an embodiment, the image presentation time determining device 50further includes a calculation unit 504 and a fusion unit 505.

The calculation unit 504 is configured to process the image sequence byusing a computer vision algorithm, to obtain a computer vision signalcorresponding to each image in the image sequence.

The obtaining unit 501 is further configured to obtain a feedback signalthat is generated when an observation object watches the plurality ofimages displayed in the presentation time sequence and that correspondsto each image in the plurality of images, where the feedback signal isused to indicate a reaction of the observation object to the watchedimage.

The fusion unit 505 is configured to fuse, for each image in the imagesequence, a corresponding computer vision signal and a correspondingfeedback signal to obtain a target recognition signal of each image inthe plurality of images, where the target recognition signal is used forimage recognition.

It should be noted that, the image presentation time determining device50 may be the image recognition device 40 in the system architectureshown in FIG. 1. It should be noted that, for implementation of eachunit in the image presentation time determining device 50, refer tocorresponding descriptions in the method embodiment shown in FIG. 4.Details are not described herein again.

FIG. 9 is a schematic structural diagram of an image presentation timeadjustment device 60 according to an embodiment of this application. Asshown in FIG. 9, the device may include an obtaining unit 601, anadjustment unit 602, and a control unit 603.

The obtaining unit 601 is configured to obtain an image sequence basedon a rapid serial visual presentation RSVP paradigm, where the imagesequence includes a plurality of images, a presentation time isconfigured for each image in the plurality of images, a presentationtime of an image i is used to indicate a time period from a presentationstart moment of the image i to a presentation start moment of a nextadjacent image, and the image i is any image in the plurality of images.

The adjustment unit 602 is configured to adjust the presentation timecorresponding to each image in the image sequence based on acorresponding duration impact parameter for each image in the imagesequence, where the duration impact parameter includes at least one of afirst recognition probability and a fatigue state parameter, the firstrecognition probability is used to indicate a probability, obtained byusing a computer vision algorithm, that an image includes a preset imagefeature, the fatigue state parameter is used to indicate a fatiguedegree of the observation object when the observation object observes animage, the first recognition probability is inversely correlated withthe presentation time, and the fatigue state parameter is positivelycorrelated with the presentation time.

The control unit 603 is configured to control display of the imagesequence based on an adjusted presentation time corresponding to eachimage in the image sequence.

In an embodiment, presentation times of the plurality of images areequal before the presentation times start to be adjusted.

In an embodiment, the obtaining unit 601 is specifically configured to:receive M images from a camera device, where M is an integer greaterthan 1; and select N images from the M images as the image sequence,where N is less than or equal to M.

In an embodiment, a fatigue state parameter corresponding to each imagein the image sequence is obtained through prediction according to afatigue rule, and the fatigue rule is used to indicate a change rule ofa fatigue degree of the observation object.

In an embodiment, a fatigue state parameter corresponding to an image isa fatigue state parameter of the observation object when the observationobject observes the image.

In an embodiment, the image presentation time adjustment device 60further includes a fusion unit 604 and a calculation unit 605. Theobtaining unit 601 is configured to obtain a feedback signal that isgenerated when the observation object watches the image sequencedisplayed in the presentation time sequence and that corresponds to eachimage in the image sequence, where the feedback signal is used toindicate a reaction of the observation object to the watched image.

The calculation unit 605 is configured to process the image sequence byusing a computer vision algorithm, to obtain a computer vision signalcorresponding to each image in the image sequence.

The fusion unit 604 is configured to fuse, for each image in the imagesequence, a corresponding computer vision signal and a correspondingfeedback signal to obtain a target recognition signal of each image inthe image sequence, where the target recognition signal is used forimage recognition.

In an embodiment, the duration impact parameter includes the firstrecognition probability and the fatigue state parameter, and the imagepresentation time adjustment device 60 further includes a predictionunit 606.

The calculation unit 605 is specifically configured to process the imagesequence by using the computer vision algorithm, to obtain the firstrecognition probability corresponding to each image in the imagesequence.

The prediction unit 606 is configured to predict a fatigue stateparameter corresponding to each image in the image sequence according toa fatigue rule, where the fatigue rule is used to indicate a change ruleof a fatigue degree of the observation object.

The adjustment unit 602 is specifically configured to adjust thepresentation time corresponding to each image in the image sequencebased on a corresponding first recognition probability and fatigue stateparameter, so as to obtain an adjusted presentation time sequencecorresponding to the image sequence.

In an embodiment, the duration impact parameter includes the firstrecognition probability. The calculation unit 605 is further configuredto process the image sequence by using the computer vision algorithm, toobtain a first recognition probability of each image in the imagesequence. The adjustment unit 602 is specifically configured to adjust,for each image in the image sequence based on a corresponding firstrecognition probability, the presentation time corresponding to eachimage in the image sequence.

In an embodiment, the duration impact parameter includes the fatiguestate parameter. The prediction unit 606 is configured to predict afatigue state parameter corresponding to each image in the imagesequence according to a fatigue rule, where the fatigue rule is used toindicate a change rule of a fatigue degree of the observation object.The adjustment unit 602 is specifically configured to adjust, for eachimage in the image sequence based on a corresponding fatigue stateparameter, the presentation time corresponding to each image in theimage sequence.

In an embodiment, the adjustment unit 602 is specifically configured tofind, for an image j, a presentation time offset of the image j from athird mapping table based on a duration impact parameter of the image j,where the third mapping table includes a plurality of duration impactparameters and presentation time offsets respectively corresponding tothe plurality of duration impact parameters; and adjust a presentationtime of the image j based on the presentation time offset of the imagej, where the image j is any image in the image sequence.

In an embodiment, the obtaining unit 601 is further configured to: foran image q, obtain a fatigue state parameter of the image q based onfatigue state information that is sent by a sensor and that is obtainedwhen the observation object watches the image p, where the image q isany image in the image sequence other than the first image, and theimage p is a previous image of the image q.

In an embodiment, a fatigue state parameter of the first image is presetto 0.

In an embodiment, the duration impact parameter includes the firstrecognition probability, and the adjustment unit 602 is specificallyconfigured to obtain a presentation time offset of each image in theimage sequence by using the following fitting formula:

${{\Delta{T(c)}} = {\sum\limits_{t = 0}^{n}{a_{j}c^{t}}}};$where

ΔT(c) is the presentation time offset, c is the first recognitionprobability, c is a real number satisfying 0≤c≤1, n is an order at whichΔT(c) fits c, n is an integer greater than 0, t is an integer satisfying−n≤t≤n, and a_(t) is a coefficient of c^(t); adjust the presentationtime of each image in the image sequence based on the presentation timeoffset of each image in the image sequence; and adjust the presentationtime corresponding to each image in the image sequence based on thecorresponding presentation time offset of each image in the imagesequence.

In an embodiment, ΔT(c) is obtained after n-order linear fitting isperformed on c by using (c1, T2) and (c2, T1). T1 is a minimumpresentation time threshold, T2 is a maximum presentation timethreshold, c1 is a minimum probability threshold of a recognitionprobability determined by using the computer vision algorithm, and c2 isa maximum probability threshold of a recognition probability determinedby using the computer vision algorithm.

In an embodiment, when a first recognition probability of an image q isgreater than or equal to c2, the first recognition probability is usedto determine that the image q includes the preset image feature. Whenthe first recognition probability of the image q is less than or equalto c1, the first recognition probability is used to determine that theimage q does not include the preset image feature, where the image q isany image in the image sequence.

In an embodiment, the duration impact parameter includes the fatiguestate parameter, and the adjustment unit 602 is specifically configuredto obtain the presentation time offset of each image in the imagesequence by using the following fitting formula:

${{\Delta{T(f)}} = {\sum\limits_{k = 0}^{m}{a_{k}f^{k}}}};$where

ΔT(f) is the presentation time offset, f is the fatigue state parameter,m is an order at which ΔT(f) fits f, m is a positive integer greaterthan 0, k is an integer satisfying −m≤k≤m, and a_(k) is a coefficient off^(k); adjust the presentation time corresponding to each image in theimage sequence based on the presentation time offset of thecorresponding image; and

adjust the presentation time corresponding to each image in the imagesequence based on the corresponding presentation time offset of eachimage in the image sequence.

In an embodiment, the duration impact parameter includes the firstrecognition probability and the fatigue state parameter, and theadjustment unit 602 is specifically configured to obtain thepresentation time offset of each image in the image sequence by usingthe following fitting formula:

${{\Delta{T\left( {c,f} \right)}} = {\sum\limits_{t = 0}^{n}{\sum\limits_{k = 0}^{m}{a_{t,k}c^{t}f^{k}}}}};$where

ΔT(c, f) is the presentation time offset, c is the first recognitionprobability, f is the fatigue state parameter, m is an order at whichΔT(c, f) fits f, n is an order at which ΔT(c, f) fits c, both n and mare positive integers greater than 0, t is an integer satisfying −n≤t≤n,k is an integer satisfying −m≤k≤m, c is a real number satisfying 0≤c≤1,and a_(t,k) is a coefficient of c^(t)f^(k); and

adjust the presentation time corresponding to each image in the imagesequence based on the corresponding presentation time offset of eachimage in the image sequence.

In an embodiment, the control unit 603 is further configured to: when itis detected that a corresponding fatigue state parameter obtained whenthe observation object observes an image r is greater than or equal to afirst fatigue threshold, control to stop displaying images to bedisplayed after the image r in the image sequence.

The obtaining unit 601 is further configured to obtain an image whosecorresponding first recognition probability is greater than or equal toa first probability threshold in the images to be displayed after theimage r.

The detection unit 603 is further configured to: when it is detectedthat the fatigue state parameter of the observation object is less thanor equal to a second fatigue threshold, control to sequentially displaythe image whose first recognition probability is greater than or equalto the first probability threshold in the images to be displayed afterthe image r, where the image r is any image in the image sequence.

In an embodiment, there are at least two observation objects, and thefatigue state parameter is at least two fatigue state parametersrespectively generated when the at least two observation objects observea same image. A presentation time of an image u is positively correlatedwith a weighted sum of the at least two fatigue state parameters, wherethe image u is any image in the image sequence. In an embodiment, foreach image in the image sequence, the fusion unit 604 is specificallyconfigured to: determine, for each image in the image sequence based onat least one of the first recognition probability, the fatigue stateparameter, and the presentation time, a first weight corresponding toeach image in the image sequence, where the first weight is a weightused when the corresponding feedback signal is used to determine thetarget recognition signal, the first weight is inversely correlated withthe first recognition probability, the first weight is inverselycorrelated with the fatigue state parameter, and the first weight ispositively correlated with the presentation time.

The fusion unit 604 is further configured to fuse, for each image in theimage sequence based on a corresponding first weight, a correspondingcomputer vision signal and a corresponding feedback signal to obtain thetarget recognition signal of each image in the image sequence.

In an embodiment, the computer vision signal is a first recognitionprobability determined by using the computer vision algorithm. Thecalculation unit 605 is further configured to calculate, for each imagein the image sequence, a second recognition probability of each image inthe image sequence based on a corresponding feedback signal, where thesecond recognition probability is used to indicate a probability thatthe observation object determines that the image includes the presetimage feature. The fusion unit 604 is specifically configured tocalculate, for each image in the image sequence, a target recognitionprobability of each image in the image sequence based on thecorresponding first recognition probability and the corresponding secondrecognition probability.

In an embodiment, the computer vision signal is an image featuredetermined by using the computer vision algorithm. The calculation unit605 is further configured to determine, for each image in the imagesequence based on a corresponding feedback signal, a feedback signalfeature corresponding to each image in the image sequence.

The fusion unit 604 is specifically configured to perform, for eachimage in the image sequence, feature fusion on the corresponding imagefeature and the corresponding feedback signal feature, to obtain a fusedfeature corresponding to each image in the image sequence.

The fusion unit 604 is further configured to determine, for each imagein the image sequence, a target recognition probability of each image inthe image sequence based on the corresponding fused feature.

In an embodiment, the image presentation time adjustment device 60further includes a determining unit 607, configured to determine, fromthe image sequence based on the target recognition probability of eachimage in the image sequence, S images as images including the presetimage feature, where the target recognition probabilities of the Simages meet a preset condition, and S is an integer less than or equalto N. The preset condition may be that the target recognitionprobability is greater than or equal to a threshold, or the presetcondition may be that the S images are the first S images sorted indescending order according to the target recognition probabilities whentarget recognition probabilities of images in the image sequence aresorted in descending order.

In an embodiment, the determining unit 607 is further configured to useimages that are in the image sequence and whose corresponding targetrecognition probabilities are between a second probability threshold anda third probability threshold as a new image sequence. The new imagesequence may be used to re-execute the method described in FIG. 6.

When a target recognition probability of any image in the image sequenceis less than or equal to the second probability threshold, the image isnot an image that includes the preset image feature. When a targetrecognition probability of any image in the image sequence is greaterthan or equal to the third probability threshold, the image is an imagethat includes the preset image feature. The second probability thresholdis less than or equal to the third probability threshold.

In an embodiment, the feedback signal is an electroencephalogram signal.

It should be noted that, for implementation of the foregoing units,refer to corresponding descriptions in the method embodiment shown inFIG. 6. Details are not described herein again. The image presentationtime adjustment device 60 may be the image recognition device 40 in thesystem architecture shown in FIG. 1.

FIG. 10 is a schematic structural diagram of an image recognition device70 according to an embodiment of this application. As shown in FIG. 10,the device may include a setting unit 701, an obtaining unit 702, and acalculation unit 703.

The setting unit 701 is configured to set a presentation time of atarget image based on a duration impact parameter of the target image,where the presentation time of the target image is used to indicate atime period from a presentation start moment of the target image to apresentation start moment of a next adjacent image, the duration impactparameter includes at least one of a first recognition probability and afatigue state parameter, the first recognition probability is used toindicate a probability, obtained by using a computer vision algorithm,that an image includes a preset image feature, the fatigue stateparameter is used to indicate a fatigue degree of an observation objectwhen the observation object observes an image, the presentation time isinversely correlated with the first recognition probability, and thepresentation time is positively correlated with the fatigue stateparameter.

The obtaining unit 702 is configured to obtain a feedback signalgenerated when the observation object observes the target image withinthe presentation time of the target image.

The calculation unit 703 is configured to determine a target recognitionprobability of the target image based on a computer vision signal andthe feedback signal of the target image, where the computer visionsignal is the first recognition probability or an image feature that isof the target image and that is determined by using the computer visionalgorithm.

It should be noted that, for implementation of the foregoing units,refer to corresponding descriptions in the method embodiment shown inFIG. 4. Details are not described herein again. The image recognitiondevice 70 may be the image recognition device 40 in the systemarchitecture shown in FIG. 1.

An embodiment of this application further provides an image recognitionsystem, including: an image recognition device 40, a display device 10,and a feedback signal collection device 20. The image recognition device40 is separately connected to the display device 10 and the feedbacksignal collection device 20. The image recognition device 40 isconfigured to execute the brain-computer combination image recognitionmethod based on image sequence presentation described in FIG. 4. Thedisplay device 10 is configured to display the image sequence, and thefeedback signal collection device 20 is configured to obtain a feedbacksignal obtained when the observation object watches any image i in theimage sequence, and feed back the feedback signal to the imagerecognition device.

Specifically, the image recognition device 40 may be the imagerecognition device described in FIG. 3 or FIG. 7. The display device 10and the feedback signal collection device 20 may be respectively thedisplay device 10 and the feedback signal collection device 20 in thesystem described in FIG. 1.

An embodiment of this application provides an image recognition system,including: an image presentation time determining device 50, a displaydevice 10, and a feedback signal collection device 20. The imagepresentation time determining device 50 is separately connected to thedisplay device 10 and the feedback signal collection device 20. Theimage presentation time determining device 50 is configured to executethe image presentation time determining method described in FIG. 4. Thedisplay device 10 is configured to display the image sequence, and thefeedback signal collection device 20 is configured to obtain a feedbacksignal obtained when the observation object watches any image i in theimage sequence, and feed back the feedback signal to the imagepresentation time determining device. The system may be the systemdescribed in FIG. 1.

Specifically, the image presentation time determining device 50 may bethe image presentation time determining device described in FIG. 8, ormay be the image recognition device described in FIG. 3. The displaydevice 10 and the feedback signal collection device 20 may berespectively the display device 10 and the feedback signal collectiondevice 20 in the system described in FIG. 1.

An embodiment of this application provides an image recognition system,including: an image presentation time adjustment device 60, a displaydevice 10, and a feedback signal collection device 20. The imagepresentation time adjustment device is separately connected to thedisplay device and the feedback signal collection device. The imagepresentation time adjustment device is configured to execute the imagerecognition method described in FIG. 6. The display device 10 isconfigured to display the image sequence, and the feedback signalcollection device 20 is configured to obtain a feedback signal obtainedwhen the observation object watches any image i in the image sequence,and feed back the feedback signal to the image presentation timeadjustment device. The system may be the system described in FIG. 1.

Specifically, the image presentation time adjustment device 60 may bethe image presentation time determining device described in FIG. 9, ormay be the image recognition device described in FIG. 3. The displaydevice 10 and the feedback signal collection device 20 may berespectively the display device 10 and the feedback signal collectiondevice 20 in the system described in FIG. 1.

An embodiment of this application provides an image recognition system,including: an image recognition device 70, a display device 10, and afeedback signal collection device 20. The image recognition device 70 isseparately connected to the display device 10 and the feedback signalcollection device 20. The image recognition device 70 is configured toexecute the image recognition method described in FIG. 6. The displaydevice 10 is configured to display the target image, and the feedbacksignal collection device 20 is configured to obtain a feedback signalobtained when the observation object observes the target image, and feedback the feedback signal to the image recognition device. The system maybe the system described in FIG. 1.

Specifically, the image recognition device 70 may be the imagerecognition device described in FIG. 10, or may be the image recognitiondevice described in FIG. 3. The display device 10 and the feedbacksignal collection device 20 may be respectively the display device 10and the feedback signal collection device 20 in the system described inFIG. 1.

An embodiment of the present invention further provides a chip system.The chip system includes at least one processor, a memory, and aninterface circuit. The memory, the transceiver, and the at least oneprocessor are interconnected by using a line, and the at least onememory stores an instruction. When the instruction is executed by theprocessor, the method procedure shown in FIG. 4 or FIG. 6 isimplemented.

An embodiment of the present invention further provides a computerreadable storage medium. The computer readable storage medium stores aninstruction, and when the instruction is run on a processor, the methodprocedure shown in FIG. 4 or FIG. 6 is implemented.

An embodiment of the present invention further provides a computerprogram product. When the computer program product is run on aprocessor, the method procedure shown in FIG. 4 or FIG. 6 isimplemented.

All or some of the foregoing embodiments may be implemented by means ofsoftware, hardware, firmware, or any combination thereof. When softwareis used to implement the embodiments, the embodiments may be implementedcompletely or partially in a form of a computer program product. Thecomputer program product includes one or more computer instructions.When the computer program instructions are loaded and executed on acomputer, some or all of the procedures or functions according to theembodiments of this application are generated. The computer may be ageneral-purpose computer, a dedicated computer, a computer network, oranother programmable apparatus. The computer instruction may be storedin a computer readable storage medium, or transmitted by using thecomputer readable storage medium. The computer instruction may betransmitted from a website, computer, server, or data center to anotherwebsite, computer, server, or data center in a wired (for example, acoaxial cable, an optical fiber, or a digital subscriber line (DSL)) orwireless (for example, infrared, radio, and microwave, or the like)manner. The computer-readable storage medium may be any usable mediumaccessible by a computer, or a data storage device, such as a server ora data center, integrating one or more usable media. The usable mediummay be a magnetic medium (for example, a floppy disk, a hard disk, or amagnetic tape), an optical medium (for example, a DVD), a semiconductormedium (for example, a solid-state drive (SSD)), or the like.

A person of ordinary skill in the art may understand that all or some ofthe processes of the methods in the embodiments may be implemented by acomputer program instructing relevant hardware. The program may bestored in a computer readable storage medium. When the program is run,the processes of the methods in the embodiments are performed. Theforegoing storage medium includes: any medium that can store programcode, such as a ROM or a random access memory RAM, a magnetic disk or anoptical disc.

What is claimed is:
 1. A brain-computer combination image recognitionmethod based on image sequence presentation, comprising: setting apresentation time sequence corresponding to an image sequence, whereinthe image sequence comprises N images, N is a positive integer, thepresentation time sequence comprises a presentation time of an image i,the presentation time of the image i is determined based on a durationimpact parameter that comprises at least one of a first recognitionprobability calculated using a computer vision algorithm or a fatiguestate parameter corresponding to the image i, and is to indicate a timeperiod from a presentation start moment of the image i to a presentationstart moment of a next adjacent image, the image i is any image in theimage sequence, the presentation time sequence comprises at least twounequal presentation times, a difference between any two presentationtimes of the at least two unequal presentation times is k×Δ, k is apositive integer, and Δ is a preset time period value; processing theimage sequence by using the computer vision algorithm, to obtain acomputer vision signal corresponding to the image i in the imagesequence; obtaining a feedback signal that is generated when anobservation object watches the image sequence displayed in thepresentation time sequence and that corresponds to the image i in theimage sequence, wherein the feedback signal is to indicate a reaction ofthe observation object to the watched image; and fusing, for the image iin the image sequence, the computer vision signal corresponding to theimage i and the feedback signal corresponding to the image i to obtain atarget recognition signal of the image i in the image sequence for imagerecognition.
 2. The method according to claim 1, wherein before thesetting of the presentation time sequence corresponding to the imagesequence, the method further comprises: receiving M images from a cameradevice, wherein M is an integer greater than 1; and selecting N imagesfrom the M images as the image sequence, wherein N is less than or equalto M.
 3. The method according to claim 1, wherein the first recognitionprobability is to indicate a probability that the image i comprises apreset image feature; wherein the fatigue state parameter is to indicatea fatigue degree of the observation object when the observation objectobserves the image i; and wherein the presentation time is inverselycorrelated with the first recognition probability, and positivelycorrelated with the fatigue state parameter.
 4. The method according toclaim 3, wherein the duration impact parameter comprises the fatiguestate parameter, and before the determining of the presentation time forthe image i in the image sequence based on the duration impactparameter, the method further comprises: predicting, according to afatigue rule, the fatigue state parameter corresponding to the image iin the image sequence, wherein the fatigue rule is to indicate a changerule of a fatigue degree of the observation object.
 5. The methodaccording to claim 3, wherein the obtaining of a feedback signal that isgenerated when the observation object watches the image sequencedisplayed in the presentation time sequence and that corresponds to theimage i in the image sequence comprises: in a process of displaying theimage sequence in the presentation time sequence, obtaining a fatiguestate parameter corresponding to an image j; adjusting, based on thefatigue state parameter corresponding to the image j, a presentationtime of the presentation time sequence, wherein the image i correspondsto an image to be displayed after the image j in the image sequence,wherein the image j is any image in the image sequence.
 6. The methodaccording to claim 5, wherein the obtaining of the fatigue stateparameter corresponding to the image j comprises: obtaining the fatiguestate parameter based on fatigue state information that is sent by asensor and that is obtained when the observation object watches theimage j.
 7. The method according to claim 3, wherein the determining ofthe presentation time for the image i in the image sequence based on theduration impact parameter comprises: for the image i in the imagesequence, finding a presentation time corresponding to the durationimpact parameter from a first mapping table, wherein the first mappingtable comprises a plurality of duration impact parameters andpresentation times respectively corresponding to the plurality ofduration impact parameters.
 8. The method according to claim 3, furthercomprising: in response detecting that the fatigue state parameterobtained when the observation object observes an image q is greater thanor equal to a first fatigue threshold, controlling to stop displayingimages to be displayed after the image q in the image sequence, andobtaining an image with a recognition probability that is greater thanor equal to a first probability threshold in the images to be displayedafter the image q, wherein the image q is any image in the imagesequence; and in response to detecting that the fatigue state parameterof the observation object is less than or equal to a second fatiguethreshold, controlling to sequentially display the image.
 9. The methodaccording to claim 3, wherein the fusing, for the image i in the imagesequence, of the computer vision signal and the feedback signal toobtain the target recognition signal of the image i in the imagesequence comprises: determining, for the image i in the image sequencebased on at least one of the first recognition probability, the fatiguestate parameter, or the presentation time, a first weight correspondingto the image i in the image sequence, wherein the first weight is aweight used when the feedback signal is to determine the targetrecognition signal, the first weight is inversely correlated with thefirst recognition probability, the first weight is inversely correlatedwith the fatigue state parameter, and the first weight is positivelycorrelated with the presentation time; and fusing, for the image i inthe image sequence based on a corresponding first weight, the computervision signal and a the feedback signal to obtain the target recognitionsignal of the image i in the image sequence.
 10. The method according toclaim 1, wherein the fusing, for the image i in the image sequence, ofthe computer vision signal and the feedback signal to obtain the targetrecognition signal of the image i in the image sequence comprises:fusing, for the image i in the image sequence, the computer visionsignal, the feedback signal, and at least one additional feedback signalcorresponding to the image to obtain the target recognition signal ofthe image i in the image sequence.
 11. The method according to claim 10,wherein the fatigue state parameter comprises at least two fatigue stateparameters respectively generated when at least two observation objectsobserve a same image.
 12. The method according to claim 1, wherein thecomputer vision signal is the first recognition probability determinedby using the computer vision algorithm; before the fusing, for the imagei in the image sequence, of the computer vision signal and the feedbacksignal to obtain the target recognition signal of the image i in theimage sequence, the method further comprises: calculating, for the imagei in the image sequence, a second recognition probability of the image iin the image sequence based on the feedback signal, wherein the secondrecognition probability is to indicate a probability that theobservation object determines that the image i comprises the presetimage feature; and wherein the fusing, for the image i in the imagesequence, the computer vision signal and the feedback signal to obtainthe target recognition signal of the image i in the image sequencecomprises: calculating, for the image i in the image sequence, a targetrecognition probability of the image i in the image sequence based onthe first recognition probability and the second recognitionprobability.
 13. An image recognition device, comprising: a processor; amemory; and an interface circuit, wherein the memory, the interfacecircuit and the processor are interconnected, wherein the memory storesprogram instructions, which, when executed by the processor, cause theprocessor to: set a presentation time sequence corresponding to an imagesequence, wherein the image sequence comprises N images, N is a positiveinteger, the presentation time sequence comprises a presentation time ofan image i, the presentation time of the image i is determined based ona duration impact parameter that comprises at least one of a firstrecognition probability calculated using a computer vision algorithm ora fatigue state parameter corresponding to the image i, and is toindicate a time period from a presentation start moment of the image ito a presentation start moment of a next adjacent image, the image isany image in the image sequence, the presentation time sequencecomprises at least two unequal presentation times, a difference betweenany two presentation times of the at least two unequal presentationtimes is k×Δ, k is a positive integer, and Δ is a preset time periodvalue, process the image sequence by using the computer visionalgorithm, to obtain a computer vision signal corresponding to the imagei in the image sequence, cause the interface circuit to obtain afeedback signal that is generated when an observation object watches theimage sequence displayed in the presentation time sequence and thatcorresponds to the image i in the image sequence, wherein the feedbacksignal is to indicate a reaction of the observation object to thewatched image, and fuse, for the image i in the image sequence, thecomputer vision signal and the feedback signal to obtain the targetrecognition signal of the image i in the image sequence for imagerecognition.
 14. The image recognition device according to claim 13,wherein: the interface circuit is configured to receive M images from acamera device, wherein M is an integer greater than 1; and the processorfurther is configured to select N images from the M images as the imagesequence, wherein N is less than or equal to M.
 15. The imagerecognition device according to claim 13, wherein the first recognitionprobability is to indicate a probability that the image i comprises apreset image feature, the fatigue state parameter is to indicate afatigue degree of the observation object when the observation objectobserves the image i, the presentation time is inversely correlated withthe first recognition probability, and the presentation time ispositively correlated with the fatigue state parameter.
 16. The imagerecognition device according to claim 15, wherein the duration impactparameter comprises the fatigue state parameter, and the imagerecognition device further comprises a prediction unit; and wherein theprocessor is configured to predict, according to a fatigue rule, thefatigue state parameter corresponding to the image i in the imagesequence, wherein the fatigue rule is to indicate a change rule of afatigue degree of the observation object.
 17. The image recognitiondevice according to claim 15, wherein the interface circuit is furtherconfigured to: in a process of displaying the image sequence in thepresentation time sequence, obtain a fatigue state parametercorresponding to an image j; and adjust, based on the fatigue stateparameter corresponding to the image j, a presentation time that is inthe presentation time sequence, wherein the image i corresponds to animage to be displayed after the image j in the image sequence, whereinthe image j is any image in the image sequence.
 18. The imagerecognition device according to claim 17, wherein the processor isconfigured to obtain the fatigue state parameter based on fatigue stateinformation that is sent by a sensor and that is obtained when theobservation object watches the image j.
 19. The image recognition deviceaccording to claim 15, wherein the processor is configured to: for theimage i in the image sequence, find a presentation time corresponding tothe duration impact parameter from a first mapping table, wherein thefirst mapping table comprises a plurality of duration impact parametersand presentation times respectively corresponding to the plurality ofduration impact parameters.
 20. A non-transitory computer readablestorage medium, wherein the storage medium is configured to storeprogram instructions, which, when executed by a processor, cause theprocessor to perform operations of recognizing a brain-computercombination image based on image sequence presentation, the operationscomprising: setting a presentation time sequence corresponding to animage sequence, wherein the image sequence comprises N images, N is apositive integer, the presentation time sequence comprises apresentation time of an image i, the presentation time of the image i isdetermined based on a duration impact parameter that comprises at leastone of a first recognition probability calculated using a computervision algorithm or a fatigue state parameter corresponding to the imagei, and is to indicate a time period from a presentation start moment ofthe image i to a presentation start moment of a next adjacent image, theimage i is any image in the image sequence, the presentation timesequence comprises at least two unequal presentation times, a differencebetween any two presentation times of the at least two unequalpresentation times is k×Δ, k is a positive integer, and Δ is a presettime period value; processing the image sequence by using the computervision algorithm, to obtain a computer vision signal corresponding tothe image i in the image sequence; obtaining a feedback signal that isgenerated when an observation object watches the image sequencedisplayed in the presentation time sequence and that corresponds to theimage i in the image sequence, wherein the feedback signal is toindicate a reaction of the observation object to the watched image; andfusing, for the image i in the image sequence, the computer visionsignal corresponding to the image i and the feedback signalcorresponding to the image i to obtain a target recognition signal ofthe image i in the image sequence for image recognition.